Prometheus Metrics: Expose OpenClaw Diagnostics

OpenClaw exposes diagnostic metrics through the official diagnostics-prometheus plugin. It listens to trusted diagnostics and internally tagged dispatcher-owned diagnostic events (queue, memory, and session-recovery signals) and serves a Prometheus text endpoint at:

GET /api/diagnostics/prometheus

The content type is text/plain; version=0.0.4; charset=utf-8, the standard Prometheus exposition format.

Warning

This route uses Gateway authentication (operator scope, trusted-operator surface). Do not expose it as a public unauthenticated /metrics endpoint. Scrape it through the same auth path you use for other operator APIs.

For traces, logs, OTLP push, and OpenTelemetry GenAI semantic attributes, see OpenTelemetry export.

Quick start

Install the plugin

openclaw plugins install clawhub:@openclaw/diagnostics-prometheus

Enable the plugin

Config

{
  plugins: {
    allow: ["diagnostics-prometheus"],
    entries: {
      "diagnostics-prometheus": { enabled: true },
    },
  },
  diagnostics: {
    enabled: true,
  },
}

CLI

openclaw plugins enable diagnostics-prometheus

Restart the Gateway

The HTTP route registers at plugin startup, so reload after enabling.

Scrape the protected route

Send the same gateway auth your operator clients use:

curl -H "Authorization: Bearer $OPENCLAW_GATEWAY_TOKEN" \
  http://127.0.0.1:18789/api/diagnostics/prometheus

Wire Prometheus

# prometheus.yml
scrape_configs:
  - job_name: openclaw
    scrape_interval: 30s
    metrics_path: /api/diagnostics/prometheus
    authorization:
      credentials_file: /etc/prometheus/openclaw-gateway-token
    static_configs:
      - targets: ["openclaw-gateway:18789"]

Note

diagnostics.enabled defaults to true; set it to false only in tightly constrained environments. If it is false, the plugin still registers the HTTP route, but no diagnostic events reach the exporter, so the response is empty.

Metrics exported

For model-call metrics, observation_unit="request" measures one observable provider request. observation_unit="turn" measures a synthetic Claude Code or Codex CLI agent turn that can contain multiple hidden provider requests. Keep those series separate when comparing latency.

Label policy

Bounded, low-cardinality labels

Prometheus labels stay bounded and low-cardinality. The exporter does not emit raw diagnostic identifiers such as runId, sessionKey, sessionId, callId, toolCallId, message IDs, chat IDs, or provider request IDs.

Label values are redacted and must match OpenClaw's low-cardinality character policy. Values that fail the policy are replaced with unknown, other, or none, depending on the metric. Labels that look like scoped agent session keys are also replaced with unknown.

Series cap and overflow accounting

The exporter caps retained time series in memory at 2048 series across counters, gauges, and histograms combined. New series beyond that cap are dropped, and openclaw_prometheus_series_dropped_total increments by one each time.

Watch this counter as a hard signal that an attribute upstream is leaking high-cardinality values. The exporter never lifts the cap automatically; if it climbs, fix the source rather than disabling the cap.

What never appears in Prometheus output

• prompt text, response text, tool inputs, tool outputs, system prompts
• Talk transcripts, audio payloads, call ids, room ids, handoff tokens, turn ids, and raw session ids
• raw provider request IDs (only bounded hashes, where applicable, on spans, never on metrics)
• session keys and session IDs
• hostnames, file paths, secret values

PromQL recipes

# Tokens per minute, split by provider
sum by (provider) (rate(openclaw_model_tokens_total[1m]))

# Spend (USD) over the last hour, by model
sum by (model) (increase(openclaw_model_cost_usd_total[1h]))

# 95th percentile model run duration
histogram_quantile(
  0.95,
  sum by (le, provider, model)
    (rate(openclaw_run_duration_seconds_bucket[5m]))
)

# Queue wait time SLO (95p under 2s)
histogram_quantile(
  0.95,
  sum by (le, lane) (rate(openclaw_queue_lane_wait_seconds_bucket[5m]))
) < 2

# Skill usage, split by bounded source
sum by (skill, source) (increase(openclaw_skill_used_total[24h]))

# Dropped Prometheus series (cardinality alarm)
increase(openclaw_prometheus_series_dropped_total[15m]) > 0

Tip

Prefer gen_ai_client_token_usage for cross-provider dashboards: it follows the OpenTelemetry GenAI semantic conventions and is consistent with metrics from non-OpenClaw GenAI services.

Choosing between Prometheus and OpenTelemetry export

OpenClaw supports both surfaces independently. You can run either, both, or neither.

diagnostics-prometheus

• Pull model: Prometheus scrapes /api/diagnostics/prometheus.
• No external collector required.
• Authenticated through normal Gateway auth.
• Surface is metrics only (no traces or logs).
• Best for stacks already standardized on Prometheus + Grafana.

diagnostics-otel

• Push model: OpenClaw sends OTLP/HTTP to a collector or OTLP-compatible backend.
• Surface includes metrics, traces, and logs.
• Bridges to Prometheus through an OpenTelemetry Collector (prometheus or prometheusremotewrite exporter) when you need both.
• See OpenTelemetry export for the full catalog.

Troubleshooting

Empty response body

• Check that diagnostics.enabled is not set to false in config (it defaults to true).
• Confirm the plugin is enabled and loaded with openclaw plugins list --enabled.
• Generate some traffic; counters and histograms only emit lines after at least one event.

401 / unauthorized

The endpoint requires the Gateway operator scope (auth: "gateway" with gatewayRuntimeScopeSurface: "trusted-operator"). Use the same token or password Prometheus uses for any other Gateway operator route. There is no public unauthenticated mode.

openclaw_prometheus_series_dropped_total is climbing

A new attribute is exceeding the 2048-series cap. Inspect recent metrics for an unexpectedly high-cardinality label and fix it at the source. The exporter intentionally drops new series instead of silently rewriting labels.

Prometheus shows stale series after a restart

The plugin keeps state in memory only. After a Gateway restart, counters reset to zero and gauges restart at their next reported value. Use PromQL rate() and increase() to handle resets cleanly.

Related

• Diagnostics export, local diagnostics zip for support bundles
• Health and readiness, /healthz and /readyz probes
• Logging, file-based logging
• OpenTelemetry export, OTLP push for traces, metrics, and logs