OpenTelemetry Export: Sending OpenClaw Diagnostics

OpenClaw sends diagnostics to external systems through the official diagnostics-otel plugin, which relies on OTLP/HTTP (protobuf) as its transport. Log records can alternatively be directed to stdout as JSONL, which suits container and sandbox log collection pipelines. Any backend or collector that understands OTLP/HTTP integrates without requiring modifications to your code. For guidance on writing logs to local files, refer to Logging.

• Diagnostics events are structured, in-process records generated by the Gateway and its bundled plugins. They cover model runs, message flow, sessions, queues, and exec.
• diagnostics-otel listens for those events and forwards them as OpenTelemetry metrics, traces, and logs via OTLP/HTTP. It can also duplicate log records to stdout in JSONL format.
• Provider calls get a W3C traceparent header from the currently active OpenTelemetry model-call span, provided the provider transport allows custom headers. Diagnostic IDs serve only as local correlation keys, and trace context originating from plugins is not forwarded.
• Exporters activate only when both the diagnostics surface and the plugin are turned on, which keeps in-process overhead essentially at zero by default.

Quick start

openclaw plugins install clawhub:@openclaw/diagnostics-otel

{
  plugins: {
    allow: ["diagnostics-otel"],
    entries: {
      "diagnostics-otel": { enabled: true },
    },
  },
  diagnostics: {
    enabled: true,
    otel: {
      enabled: true,
      endpoint: "http://otel-collector:4318",
      protocol: "http/protobuf",
      serviceName: "openclaw-gateway",
      traces: true,
      metrics: true,
      logs: true,
      sampleRate: 0.2,
      flushIntervalMs: 60000,
    },
  },
}

You can also activate the plugin through the command line: openclaw plugins enable diagnostics-otel.

Note

diagnostics.otel.protocol recognizes only http/protobuf. If a persisted configuration, including one that gets its value through ${VAR} interpolation, still leaves this field set to the deprecated grpc value, execute openclaw doctor --fix. Doctor fixes values written directly and a single internal include that owns the top-level diagnostics section. For root or array includes, nested include chains, sibling overrides, external include targets, or any other ambiguous origin, Doctor leaves the files untouched and points out the candidate source file or files for manual editing.

When diagnostics.otel.protocol is not set, each plugin-owned OTLP signal first consults its nonblank OTEL_EXPORTER_OTLP_*_PROTOCOL value, then OTEL_EXPORTER_OTLP_PROTOCOL, and finally falls back to http/protobuf. Doctor does not modify process environment variables. A value that is not supported disables only that particular plugin-owned OTLP signal; supported sibling signals keep working, and so does the stdout branch of logsExporter: "both". Preloaded trace and metric SDKs manage their own transport selection and are not subject to rejection by this plugin.

Signals exported

Switch traces, metrics, and logs on or off separately. Traces and metrics start enabled whenever diagnostics.otel.enabled is true; logs start disabled and emit only when diagnostics.otel.logs is set explicitly to true. Log export uses OTLP by default; change diagnostics.otel.logsExporter to stdout for JSONL on stdout, or both to get both.

Note

The shared endpoint and OTEL_EXPORTER_OTLP_ENDPOINT act as the base for every enabled signal. OpenClaw attaches /v1/traces, /v1/metrics, or /v1/logs to root and custom collector paths. For hosted frontends that need compatibility, a shared endpoint that already terminates in one of those signal paths keeps that path for its matching signal and swaps the final segment for the other signals.

Signal-specific tracesEndpoint, metricsEndpoint, and logsEndpoint settings, along with their corresponding OTEL_EXPORTER_OTLP_*_ENDPOINT fallbacks, are handed to the exporter as exact URLs. OpenClaw does not append to or rewrite their paths.

Which processes export

• Gateway launches the exporter at startup and exports from the Gateway process for every run it handles, including openclaw agent turns routed to it.
• One-shot local runs (openclaw agent --local) execute within the CLI process. When OTel export is set up and the plugin is active, that same CLI process starts one exporter instance for the run and flushes buffered spans, metrics, and logs before exiting. The CLI waits up to 5 seconds for the diagnostic-event queue to empty and 10 more for the flush, so an unreachable collector cannot keep the command open. A collector that accepts the connection but never responds can still postpone exit until the exporter's own request timeout (OTEL_EXPORTER_OTLP_TIMEOUT) fires. In JSON output mode, these one-shot runs suppress only the stdout JSONL log sink so command stdout stays clear for the JSON response; OTLP traces, metrics, and logs keep flowing when configured.
• openclaw agent exec also runs the agent embedded in the CLI process, but it does not start this exporter yet, so its runs produce no telemetry. Route dispatch through the Gateway, or use openclaw agent --local, when you need traces from a headless run.

Exporter health

openclaw doctor and openclaw status --all present a bounded, redacted view of the running Gateway's most recent trusted exporter state for each signal and transport. For diagnostics-otel, the snapshot distinguishes:

• OTLP/HTTP protobuf with an endpoint from config or an OTEL_* environment fallback.
• OTLP/HTTP protobuf using the exporter dependency's default endpoint because no endpoint was provided.
• Stdout log export.
• Trace or metric export owned by an externally preloaded OpenTelemetry SDK.

OTLP export failure and recovery transitions are captured from the exporter's final result callback, after dependency-owned retries finish. A retryable response that later succeeds is therefore not reported as a failure. Startup, log preparation or emit, export, and shutdown failures use fixed reason categories rather than raw errors.

The snapshot never includes endpoint values, headers, certificates, payloads, or raw error messages. Transport is retained only in this local health projection. It is not added to the existing openclaw.telemetry.exporter.events metric attributes, and existing Prometheus label sets are unchanged.

Configuration reference

{
  diagnostics: {
    enabled: true,
    otel: {
      enabled: true,
      endpoint: "http://otel-collector:4318",
      tracesEndpoint: "http://otel-collector:4318/v1/traces",
      metricsEndpoint: "http://otel-collector:4318/v1/metrics",
      logsEndpoint: "http://otel-collector:4318/v1/logs",
      protocol: "http/protobuf",
      serviceName: "openclaw-gateway", // unset falls back to OTEL_SERVICE_NAME, then "openclaw"
      metricNamePrefix: "acme.", // optional; include the separator
      headers: { "x-collector-token": "..." },
      traces: true,
      metrics: true,
      logs: true,
      logsExporter: "otlp", // otlp | stdout | both
      sampleRate: 0.2, // root-span sampler, 0.0..1.0
      flushIntervalMs: 60000, // metric export interval (min 1000ms)
      captureContent: false,
    },
  },
}

metricNamePrefix only swaps out the default openclaw. prefix for metrics that OpenClaw itself generates. As an example, "acme." turns openclaw.tokens into acme.tokens; choosing "" makes tokens appear with no prefix at all. Any non-empty value must begin with an ASCII letter, contain solely letters, digits, underscores, dots, hyphens, and slashes, and stay within 128 characters. Pick "acme.openclaw." when you need acme.openclaw.tokens. Metrics that follow standard semantic conventions, like gen_ai.client.token.usage and gen_ai.client.operation.duration, are left untouched. If you leave this option unset, all current metric names stay as they are. Because enabling or adjusting this setting renames the affected metric series, any dashboards, alerts, or recording rules relying on the previous names must be updated.

Environment variables

Without OPENCLAW_OTEL_PRELOADED=1, trace, metric, and log providers stay private to their generation. Only the async context manager and propagator are published through the public OpenTelemetry APIs, and they are removed solely when those public behaviors still match the generation being stopped. A replacement host or later generation therefore keeps ownership through cleanup.

Continue an upstream WebSocket trace

An authenticated Gateway WebSocket client can attach a W3C traceparent to each request frame:

{
  "type": "req",
  "id": "eval-item-42",
  "method": "agent",
  "params": {},
  "traceparent": "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01"
}

A child request context is created by the Gateway, preserving the upstream trace ID and sampling flags. Spans for agents, harnesses, model calls, providers, tool executions, and execs created within the request remain on that trace, even those recorded after the parent run has completed. This lets a local experiment runner create one Langfuse/OpenTelemetry trace per dataset item and correlate the corresponding OpenClaw execution.

Trace context applies per request, not per connection. On a long-lived WebSocket, generate or inject the appropriate traceparent independently for every RPC. Concurrent requests stay isolated even when their work interleaves.

The field is accepted only after the existing Gateway authentication handshake and does not affect authentication or method authorization. A traceparent on the initial connect frame is ignored. Missing or syntactically malformed values within the 128-character field limit silently fall back to a fresh request trace; longer values make the request frame invalid. tracestate and baggage are not accepted by the Gateway WebSocket protocol.

Privacy and content capture

Raw model/tool content is not exported by default. Spans carry bounded identifiers (channel, provider, model, error category, hash-only request ids, tool source, tool owner, skill name/source) and never include prompt text, response text, tool inputs, tool outputs, skill file paths, or session keys. Values that look like scoped agent session keys (for example starting with agent:) are replaced with unknown on low-cardinality attributes. OTLP log records keep severity, logger, code location, trusted trace context, and sanitized attributes by default; the raw log message body is exported only when diagnostics.otel.captureContent is true. Talk metrics export only bounded event metadata (mode, transport, provider, event type) - no transcripts, audio payloads, session ids, turn ids, call ids, room ids, or handoff tokens.

When diagnostics-otel tracing is active, outbound model requests may include a W3C traceparent header from the actual exporter-owned model-call span. Diagnostic trace IDs and span IDs only correlate events to that span; they are not used as outbound OTel identities. If the exporter cannot resolve a real span context, OpenClaw omits the header instead of naming an unexported parent. Existing caller-supplied traceparent headers are removed or replaced, so plugins or custom provider options cannot spoof cross-service trace ancestry.

Set diagnostics.otel.captureContent to true only when your collector and retention policy are approved for prompt, response, tool, and tool-definition text. This enables bounded, redacted input messages, output messages, tool inputs, tool outputs, tool definitions, and OTLP log bodies. System prompts remain excluded. Provider-internal thinking and redacted_thinking payloads are also excluded: compatibility attributes retain only a redacted structural marker, while GenAI message attributes omit those parts.

toolInputs/toolOutputs content is captured for the built-in agent runtime's tool executions (openclaw.content.tool_input and gen_ai.tool.call.arguments on completed/error spans; openclaw.content.tool_output and gen_ai.tool.call.result on completed spans). The openclaw.content.* names remain the stable OpenClaw attribute names; the gen_ai.tool.call.* copies mirror them for semconv-native viewers. External harness tool calls (Codex, Claude CLI) emit tool.execution.* spans without content payloads. Captured content travels on a trusted, listener-only channel and is never placed on the public diagnostic event bus.

Sampling and flushing

• Traces: diagnostics.otel.sampleRate sets a TraceIdRatioBasedSampler on the root span only (0.0 drops all, 1.0 keeps all). Unset uses the OpenTelemetry SDK default (always-on).
• Metrics: diagnostics.otel.flushIntervalMs (clamped to a minimum of 1000); unset uses the SDK's periodic-export default.
• Logs: OTLP logs respect logging.level (file log level) and use the diagnostic log-record redaction path, not console formatting. High-volume installs should prefer OTLP collector sampling/filtering over local sampling. Set diagnostics.otel.logsExporter: "stdout" when your platform already ships stdout/stderr to a log processor and you have no OTLP logs collector. Stdout records are one JSON object per line with ts, signal, service.name, severity, body, redacted attributes, and trusted trace fields when available.
• File-log correlation: JSONL file logs include top-level traceId, spanId, parentSpanId, and traceFlags when the log call carries a valid diagnostic trace context, letting log processors join local log lines with exported spans.
• Request correlation: Gateway HTTP requests and WebSocket frames create an internal request trace scope. Logs and diagnostic events inside that scope inherit the request trace by default, while agent run and model-call spans are created as children so provider traceparent headers stay on the same trace.
• Model-call correlation: openclaw.model.call spans include safe prompt component sizes by default and per-call token attributes when the provider result exposes usage. openclaw.model.usage remains the run-level accounting span for aggregate cost, context, and channel dashboards, and stays on the same diagnostic trace when the emitting runtime has trusted trace context.

Model-call observation units

Every openclaw.model.call span identifies what its lifecycle measures through openclaw.model_call.observation_unit:

• request - one observable model/provider request. Native embedded model calls use this unit, and exporters treat a missing value as request for compatibility with older or external emitters.
• turn - one opaque agent CLI turn that may contain hidden model requests, retries, tool work, or background work. Claude Code CLI and Codex app-server calls use this unit.

Both units are model-call spans, which lets trace backends render model input, output, usage, and hierarchy. Request spans rely on the API-derived GenAI operation (chat, generate_content, or text_completion), whereas turn spans use gen_ai.operation.name = invoke_agent. Both feed into gen_ai.client.operation.duration, where the operation name keeps direct request latency separate from full-turn latency. OpenClaw's OTEL model-call metrics additionally include openclaw.model_call.observation_unit; the Prometheus model-call metrics expose the equivalent observation_unit label.

Claude Code CLI model-call fidelity

Claude Code CLI turns emit a single synthetic, turn-level openclaw.model.call span. These are not Anthropic HTTP request spans. They use openclaw.api = claude-code, openclaw.model_call.observation_unit = turn, and mark the operation as gen_ai.operation.name = invoke_agent. They identify OpenClaw's CLI boundary via openclaw.transport:

• stdio - a one-shot local Claude Code process.
• stdio-live - one turn on a managed persistent Claude stdio session.
• paired-node-cli - one-shot Claude Code execution delegated to a paired node.

Claude CLI diagnostics are instantiated only while the process diagnostic dispatcher is enabled and an internal or trusted event listener is attached. With no observability plugin or other listener active, Claude CLI turns skip the synthetic trace hierarchy, content buffers, and diagnostic stream-byte accounting. When content capture is enabled, prompt and system-prompt fields are capped at 128 KiB each; assistant output is capped at 128 KiB across at most 200 envelopes, with 16 KiB and one item reserved for a final visible fallback response. A marker records truncation when the limit is reached.

OpenClaw gives Claude CLI turns the same ownership hierarchy used by other agent runtimes: openclaw.harness.run (openclaw.harness.id = claude-cli) contains openclaw.run, which contains the Claude openclaw.model.call span. The harness and run spans are synthetic OpenClaw turn boundaries, not Claude Code internal phases. One-shot and managed stdio turns use the same hierarchy; a real fresh-session retry creates another model-call child inside the same OpenClaw run.

The span starts when OpenClaw admits the prepared CLI turn and ends only after that turn succeeds or fails. For managed sessions, an interim success result does not end the span while Claude reports result-holding background agents or workflows; the final post-drain result does. Abort, timeout, process failure, output/parse failure, and other turn failures end the same span with an error.

Claude Code reports per-assistant-message usage and may also report cumulative usage on its terminal result. OpenClaw reply accounting continues to use the last assistant message so existing cost semantics do not change; the turn-level model-call span uses terminal cumulative usage when available, including cache-read and cache-creation tokens.

For these CLI spans, byte and timing fields describe the observable OpenClaw CLI boundary:

• openclaw.model_call.request_bytes is the UTF-8 size of the prompt value sent over one-shot stdin/argv, or the managed stdio JSONL user envelope. It is not the size of Claude Code's hidden model request.
• openclaw.model_call.response_bytes is the UTF-8 size of Claude CLI stdout observed during the turn. It is not Anthropic HTTP response size.
• openclaw.model_call.time_to_first_byte_ms is time to the first observable Claude CLI stdout or stderr output. It is not network TTFB.

With captureContent enabled, the span exports the effective prompt OpenClaw sends to Claude Code and visible assistant text/tool-call identity through gen_ai.input.messages and gen_ai.output.messages. Tool arguments, internal thinking, opaque thinking signatures, tool results, and system prompts are omitted from the Claude assistant envelope. OpenClaw does not claim access to Claude Code's private system prompt, hidden resumed or compacted request payload, native internal tool schemas, raw Anthropic HTTP request, internal retries, upstream request id, or true network TTFB. Because Claude Code does not expose its effective native tool definitions accurately, these spans do not populate gen_ai.tool.definitions.

External Claude harness tool spans remain metadata-only even when tool content capture is enabled. As with every model span, captured Claude CLI content uses the trusted listener-only path and the exporter's existing redaction and size bounds; content remains off by default.

Exported metrics

Model usage

• openclaw.tokens (counter, attrs: openclaw.token, openclaw.channel, openclaw.provider, openclaw.model, openclaw.agent)
• openclaw.cost.usd (counter, attrs: openclaw.channel, openclaw.provider, openclaw.model)
• openclaw.run.duration_ms (histogram, attrs: openclaw.channel, openclaw.provider, openclaw.model)
• openclaw.context.tokens (histogram, attrs: openclaw.context, openclaw.channel, openclaw.provider, openclaw.model)
• gen_ai.client.token.usage (histogram, GenAI semantic-conventions metric, attrs: gen_ai.token.type = input/output, gen_ai.provider.name, gen_ai.operation.name, gen_ai.request.model)
• gen_ai.client.operation.duration (histogram, seconds, GenAI semantic-conventions metric for model requests and synthetic agent turns; attrs: gen_ai.provider.name, gen_ai.operation.name, gen_ai.request.model, optional error.type; turn observations use gen_ai.operation.name = invoke_agent)
• openclaw.model_call.duration_ms (histogram, attrs: openclaw.provider, openclaw.model, openclaw.api, openclaw.transport, openclaw.model_call.observation_unit, plus openclaw.errorCategory and openclaw.failureKind on classified errors)
• openclaw.model_call.request_bytes (histogram, UTF-8 byte size of the final model request payload; for Claude Code CLI, the observable prompt input/envelope described above; no raw payload content)
• openclaw.model_call.response_bytes (histogram, UTF-8 byte size of streamed response chunk payloads; high-frequency text, thinking, and tool-call deltas count only incremental delta bytes; for Claude Code CLI, observed stdout bytes; no raw response content)
• openclaw.model_call.time_to_first_byte_ms (histogram, elapsed time before the first streamed response event; for Claude Code CLI, first observable CLI output rather than network TTFB)
• openclaw.model.failover (counter, attrs: openclaw.provider, openclaw.model, openclaw.failover.to_provider, openclaw.failover.to_model, openclaw.failover.reason, openclaw.failover.suspended, openclaw.lane)
• openclaw.skill.used (counter, attrs: openclaw.skill.name, openclaw.skill.source, openclaw.skill.activation, optional openclaw.agent, optional openclaw.toolName)

Message flow

• openclaw.webhook.received (counter, attrs: openclaw.channel, openclaw.webhook)
• openclaw.webhook.error (counter, attrs: openclaw.channel, openclaw.webhook)
• openclaw.webhook.duration_ms (histogram, attrs: openclaw.channel, openclaw.webhook)
• openclaw.message.queued (counter, attrs: openclaw.channel, openclaw.source)
• openclaw.message.received (counter, attrs: openclaw.channel, openclaw.source)
• openclaw.message.dispatch.started (counter, attrs: openclaw.channel, openclaw.source)
• openclaw.message.dispatch.completed (counter, attrs: openclaw.channel, openclaw.outcome, openclaw.reason, openclaw.source)
• openclaw.message.dispatch.duration_ms (histogram, attrs: openclaw.channel, openclaw.outcome, openclaw.reason, openclaw.source)
• openclaw.message.processed (counter, attrs: openclaw.channel, openclaw.outcome)
• openclaw.message.duration_ms (histogram, attrs: openclaw.channel, openclaw.outcome)
• openclaw.message.delivery.started (counter, attrs: openclaw.channel, openclaw.delivery.kind)
• openclaw.message.delivery.duration_ms (histogram, attrs: openclaw.channel, openclaw.delivery.kind, openclaw.outcome, openclaw.errorCategory)

Talk

• openclaw.talk.event (counter, attrs: openclaw.talk.event_type, openclaw.talk.mode, openclaw.talk.transport, openclaw.talk.brain, openclaw.talk.provider)
• openclaw.talk.event.duration_ms (histogram, attrs: same as openclaw.talk.event; emitted when a Talk event reports duration)
• openclaw.talk.audio.bytes (histogram, attrs: same as openclaw.talk.event; emitted for Talk audio frame events that report byte length)

Queues and sessions

• openclaw.queue.lane.enqueue (counter, attrs: openclaw.lane)
• openclaw.queue.lane.dequeue (counter, attrs: openclaw.lane)
• openclaw.queue.depth (histogram, attrs: openclaw.lane or openclaw.channel=heartbeat)
• openclaw.queue.wait_ms (histogram, attrs: openclaw.lane)
• openclaw.session.state (counter, attrs: openclaw.state, openclaw.reason)
• openclaw.session.stuck (counter, attrs: openclaw.state; emitted for recoverable stale session bookkeeping)
• openclaw.session.stuck_age_ms (histogram, attrs: openclaw.state; emitted for recoverable stale session bookkeeping)
• openclaw.session.turn.created (counter, attrs: openclaw.agent, openclaw.channel, openclaw.trigger)
• openclaw.session.recovery.requested (counter, attrs: openclaw.state, openclaw.action, openclaw.active_work_kind, openclaw.reason)
• openclaw.session.recovery.completed (counter, attrs: openclaw.state, openclaw.action, openclaw.status, openclaw.active_work_kind, openclaw.reason)
• openclaw.session.recovery.age_ms (histogram, attrs: same as the matching recovery counter)
• openclaw.run.attempt (counter, attrs: openclaw.attempt)

Session liveness telemetry

As long as OpenClaw sees reply, tool, status, block, or ACP runtime activity, a processing session will not drift toward the built-in liveness limit. Typing keepalives are excluded from progress, so a silent model or harness remains detectable.

OpenClaw groups sessions by what it can still observe:

• session.long_running: embedded work, model calls, or tool calls are actively progressing. Owned silent model calls also register as long-running before the built-in abort threshold, so slow or non-streaming model providers do not look like stalled gateway sessions while abort-observable.
• session.stalled: active work exists, but the active run has not shown recent progress. Owned model calls transition from session.long_running to session.stalled at or after the built-in abort threshold; ownerless stale model/tool activity is not considered harmless long-running work. Stalled embedded runs stay observe-only initially, then abort-drain after the abort threshold with no progress so queued turns behind the lane can resume.
• session.stuck: stale session bookkeeping with no active work, or an idle queued session with stale ownerless model/tool activity. This frees the affected session lane immediately after recovery gates pass.

Recovery emits structured session.recovery.requested and session.recovery.completed events. Diagnostic session state is marked idle only after a mutating recovery outcome (aborted or released) and only if the same processing generation is still current.

Only session.stuck emits the openclaw.session.stuck counter, the openclaw.session.stuck_age_ms histogram, and the openclaw.session.stuck span. Repeated session.stuck diagnostics back off while the session remains unchanged, so dashboards should alert on sustained increases rather than every heartbeat tick. For the config knob and defaults, see Configuration reference.

Liveness warnings also emit:

• openclaw.liveness.warning (counter, attrs: openclaw.liveness.reason)
• openclaw.liveness.event_loop_delay_p99_ms (histogram, attrs: openclaw.liveness.reason)
• openclaw.liveness.event_loop_delay_max_ms (histogram, attrs: openclaw.liveness.reason)
• openclaw.liveness.event_loop_utilization (histogram, attrs: openclaw.liveness.reason)
• openclaw.liveness.cpu_core_ratio (histogram, attrs: openclaw.liveness.reason)

Harness lifecycle

• openclaw.harness.duration_ms (histogram, attrs: openclaw.harness.id, openclaw.harness.plugin, openclaw.outcome, openclaw.harness.phase when errors occur)

Tool execution and loop detection

• openclaw.tool.execution.duration_ms (histogram, attrs: gen_ai.tool.name, openclaw.toolName, openclaw.tool.source, openclaw.tool.owner, openclaw.tool.params.kind, with openclaw.errorCategory added on errors)
• openclaw.tool.execution.blocked (counter, attrs: gen_ai.tool.name, openclaw.toolName, openclaw.tool.source, openclaw.tool.owner, openclaw.tool.params.kind, openclaw.deniedReason)
• openclaw.tool.loop (counter, attrs: openclaw.toolName, openclaw.loop.level, openclaw.loop.action, openclaw.loop.detector, openclaw.loop.count, optional openclaw.loop.paired_tool; triggered when a recurring tool-call pattern is identified)

Exec

• openclaw.exec.duration_ms (histogram, attrs: openclaw.exec.target, openclaw.exec.mode, openclaw.outcome, openclaw.failureKind)

Diagnostics internals (memory, payloads, exporter health)

• openclaw.payload.large (counter, attrs: openclaw.payload.surface, openclaw.payload.action, openclaw.channel, openclaw.plugin, openclaw.reason)
• openclaw.payload.large_bytes (histogram, attrs: identical to openclaw.payload.large)
• openclaw.memory.rss_bytes / openclaw.memory.heap_used_bytes / openclaw.memory.heap_total_bytes / openclaw.memory.external_bytes / openclaw.memory.array_buffers_bytes (histograms, no attrs; process memory samples)
• openclaw.memory.pressure (counter, attrs: openclaw.memory.level, openclaw.memory.reason)
• openclaw.diagnostic.async_queue.dropped (counter, attrs: openclaw.diagnostic.async_queue.drop_class; internal diagnostic-queue backpressure drops)
• openclaw.telemetry.exporter.events (counter, attrs: openclaw.exporter, openclaw.signal, openclaw.status, optional openclaw.reason, optional openclaw.errorCategory; exporter lifecycle/failure self-telemetry)

Exported spans

• openclaw.model.usage
  • openclaw.channel, openclaw.provider, openclaw.model
  • Optional host-derived openclaw.plugin only for trusted plugin runtime completions
  • openclaw.tokens.* (input/output/cache_read/cache_write/total)
  • gen_ai.system by default, or gen_ai.provider.name when the latest GenAI semantic conventions are opted in
  • gen_ai.request.model, gen_ai.operation.name, gen_ai.usage.*

Attribution for plugins applies exclusively to spans. It never adds a plugin dimension to shared OpenTelemetry metrics, nor does it alter Prometheus metric labels.

• openclaw.run
  • openclaw.outcome, openclaw.channel, openclaw.provider, openclaw.model, openclaw.errorCategory
• openclaw.model.call
  • gen_ai.system serves as the default, while gen_ai.provider.name applies when the latest GenAI semantic conventions are selected
  • gen_ai.request.model, gen_ai.operation.name, openclaw.provider, openclaw.model, openclaw.api, openclaw.transport, openclaw.model_call.observation_unit (either request or turn)
  • openclaw.errorCategory, error.type, plus openclaw.failureKind on errors when applicable
  • openclaw.model_call.request_bytes, openclaw.model_call.response_bytes, openclaw.model_call.time_to_first_byte_ms
  • openclaw.model_call.prompt.input_messages_count, openclaw.model_call.prompt.input_messages_chars, openclaw.model_call.prompt.system_prompt_chars, openclaw.model_call.prompt.tool_definitions_count, openclaw.model_call.prompt.tool_definitions_chars, openclaw.model_call.prompt.total_chars (restricted to safe component sizes, prompt text excluded)
  • openclaw.model_call.usage.* and gen_ai.usage.* appear when the response includes usage data for that specific request or the aggregate turn
  • Span event openclaw.provider.request with attribute openclaw.upstreamRequestIdHash (bounded and hash-based) fires when the upstream provider result exposes a request id; raw ids are never sent out
  • Under OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental, request spans adopt the latest GenAI inference span name {gen_ai.operation.name} {gen_ai.request.model}. Turn spans rely on invoke_agent since OpenClaw does not assert a native agent name from the opaque CLI boundary. Both employ the CLIENT span kind rather than openclaw.model.call.
• openclaw.harness.run
  • openclaw.harness.id, openclaw.harness.plugin, openclaw.outcome, openclaw.provider, openclaw.model, openclaw.channel
  • At completion: openclaw.harness.result_classification, openclaw.harness.yield_detected, openclaw.harness.items.started, openclaw.harness.items.completed, openclaw.harness.items.active
  • When an error occurs: openclaw.harness.phase, openclaw.errorCategory, and optionally openclaw.harness.cleanup_failed
• openclaw.tool.execution
• gen_ai.tool.name, gen_ai.operation.name (execute_tool), openclaw.toolName, openclaw.tool.source, plus optional gen_ai.tool.call.id, openclaw.tool.owner, openclaw.tool.params.*
• Optional openclaw.errorCategory/openclaw.errorCode for errors, openclaw.deniedReason and openclaw.outcome=blocked when policy or sandbox denies
• openclaw.exec
  • openclaw.exec.target, openclaw.exec.mode, openclaw.outcome, openclaw.failureKind, openclaw.exec.command_length, openclaw.exec.exit_code, openclaw.exec.exit_signal, openclaw.exec.timed_out
• openclaw.webhook.processed
  • openclaw.channel, openclaw.webhook
• openclaw.webhook.error
  • openclaw.channel, openclaw.webhook, openclaw.error
• openclaw.message.processed
  • openclaw.channel, openclaw.outcome, openclaw.reason
• openclaw.message.delivery
  • openclaw.channel, openclaw.delivery.kind, openclaw.outcome, openclaw.errorCategory, openclaw.delivery.result_count
• openclaw.session.stuck
  • openclaw.state, openclaw.ageMs, openclaw.queueDepth
• openclaw.context.assembled
  • openclaw.prompt.size, openclaw.history.size, openclaw.context.tokens, openclaw.errorCategory (excludes prompt, history, response, or session-key data)
• openclaw.tool.loop
  • openclaw.toolName, openclaw.loop.level, openclaw.loop.action, openclaw.loop.detector, openclaw.loop.count, optional openclaw.loop.paired_tool (loop messages, params, and tool output omitted)
• openclaw.memory.pressure
• openclaw.memory.level, openclaw.memory.reason, openclaw.memory.rss_bytes, openclaw.memory.heap_used_bytes, openclaw.memory.heap_total_bytes, openclaw.memory.external_bytes, openclaw.memory.array_buffers_bytes, optional openclaw.memory.threshold_bytes/openclaw.memory.rss_growth_bytes/openclaw.memory.window_ms

When content capture is turned on explicitly, model and tool spans may additionally carry bounded, redacted openclaw.content.* attributes for the content classes you selected.

Diagnostic event catalog

The metrics and spans described above are backed by the events listed here. Public events can be subscribed to directly by plugins; trusted core events like model.usage are limited to authorized internal consumers. run.progress and run.execution_phase serve as direct-only lifecycle signals, and the diagnostics-otel plugin does not emit them as separate OTLP signals. Event kinds and run.execution_phase.phase values are additive. TypeScript consumers should retain default branches rather than assuming either union is permanently exhaustive.

Model usage

model.usage is a trusted, in-process diagnostic event, not a JSONL log record. A representative event looks like this:

{
  "type": "model.usage",
  "ts": 1735689600000,
  "seq": 42,
  "provider": "openai",
  "model": "gpt-5.4",
  "channel": "webchat",
  "agentId": "main",
  "sessionId": "session-123",
  "sessionKey": "agent:main:main",
  "usage": {
    "input": 120,
    "output": 40,
    "cacheRead": 30,
    "cacheWrite": 10,
    "promptTokens": 160,
    "total": 200
  },
  "lastCallUsage": {
    "input": 120,
    "output": 40,
    "cacheRead": 30,
    "cacheWrite": 10,
    "total": 200
  },
  "context": { "limit": 128000, "used": 160 },
  "costUsd": 0.0012,
  "durationMs": 850,
  "trace": {
    "traceId": "4bf92f3577b34da6a3ce929d0e0e4736",
    "spanId": "00f067aa0ba902b7",
    "traceFlags": "01"
  }
}

• ts is a Unix timestamp in milliseconds; seq is process-local.
• usage holds turn-level token counts. promptTokens includes input, cacheRead, and cacheWrite; lastCallUsage, when present, describes the final model call.
• context.used is the current prompt/context snapshot and may be lower than usage.total when cached input or tool-loop calls are involved.
• Provider/model/session identifiers, token buckets, lastCallUsage, context, costUsd, durationMs, and trace fields are optional. costUsd is an estimate and can be absent when model pricing is unavailable; it is not provider-reported billing. Trace context can also include parentSpanId.

The Gateway's /tmp/openclaw/openclaw-YYYY-MM-DD.log JSONL file and diagnostics.otel.logsExporter: "stdout" contain ordinary log records, not raw model.usage events. Public diagnostic subscriptions and diagnostics.stability do not expose trusted core usage events. The diagnostics-otel plugin converts them to metrics such as openclaw.tokens and openclaw.cost.usd and to openclaw.model.usage spans; those usage metrics and spans intentionally omit session identifiers.

For an external integration that needs session-correlated usage, query the authenticated Gateway instead:

openclaw gateway call sessions.usage --params '{"range":"30d","agentScope":"all"}' --json
openclaw gateway usage-cost --days 30 --all-agents --json

Both commands require operator.read. sessions.usage can include per-session sessionId, provider/model details, and token/cost summaries; per-session usage can be temporarily null while its cache refreshes. usage-cost provides aggregate estimates. Omit agentScope or --all-agents to scope the report to the default agent. For continuously updated clients, subscribe to session changes instead of polling usage reports. See the Gateway RPC method reference for usage methods and request options.

Message flow

• webhook.received / webhook.processed / webhook.error
• message.queued / message.processed
• message.delivery.started / message.delivery.completed / message.delivery.error

Queue and session

• queue.lane.enqueue / queue.lane.dequeue
• session.state / session.long_running / session.stalled / session.stuck
• run.attempt / run.progress
• run.execution_phase (public, session-correlated embedded-runner startup milestones)
• diagnostic.heartbeat (aggregate counters: webhooks/queue/session)

Harness lifecycle

• harness.run.started / harness.run.completed / harness.run.error - per-run lifecycle for the agent harness. Includes harnessId, optional pluginId, provider/model/channel, and run id. Completion adds durationMs, outcome, optional resultClassification, yieldDetected, and itemLifecycle counts. Errors add phase (prepare/start/send/resolve/cleanup), errorCategory, and optional cleanupFailed.

Exec

• exec.process.completed - terminal outcome, duration, target, mode, exit code, and failure kind. Command text and working directories are not included.
• exec.approval.followup_suppressed - stale approval follow-up dropped after a session rebound. Includes approvalId, reason (session_rebound), phase (direct_delivery or gateway_preflight), and the dispatcher timestamp. Session keys, routes, and command text are not included.

Without an exporter

Keep diagnostics events available to plugins or custom sinks without running diagnostics-otel:

{
  diagnostics: { enabled: true },
}

For targeted debug output without raising logging.level, use diagnostics flags. Flags are case-insensitive and support wildcards (telegram.* or *):

{
  diagnostics: { flags: ["telegram.http"] },
}

Or as a one-off env override:

OPENCLAW_DIAGNOSTICS=telegram.http,telegram.payload openclaw gateway

Flag output goes to the standard log file (logging.file) and is still redacted by the always-on log redaction policy. Full guide: Diagnostics flags.

Disable

{
  diagnostics: { otel: { enabled: false } },
}

Or leave diagnostics-otel out of plugins.allow, or run openclaw plugins disable diagnostics-otel.

When the plugin would otherwise own NodeSDK, keep propagation available while disabling every plugin-owned exporter, listener, health route, and stdout sink:

OTEL_SDK_DISABLED=true openclaw gateway

Related

• Logging - file logs, console output, CLI tailing, and the Control UI Logs tab
• Gateway logging internals - WS log styles, subsystem prefixes, and console capture
• Diagnostics flags - targeted debug-log flags
• Diagnostics export - operator support-bundle tool (separate from OTEL export)
• Configuration reference - full diagnostics.* field reference