Prometheus Metrics for OpenClaw Diagnostics via diagnostics-prometheus Plugin

OpenClaw can make diagnostic metrics available through the official diagnostics-prometheus plugin. It accepts trusted diagnostics along with internally tagged dispatcher-owned diagnostic events (covering queue, memory, and session-recovery signals), then serves a Prometheus text endpoint at:

GET /api/diagnostics/prometheus

The content type is text/plain; version=0.0.4; charset=utf-8, which matches the standard Prometheus exposition format.

Warning

This route relies on Gateway authentication (operator scope, trusted-operator surface). Do not make it available as a public unauthenticated /metrics endpoint. Scrape it using the same auth path you employ for other operator APIs.

For traces, logs, OTLP push, and OpenTelemetry GenAI semantic attributes, refer to OpenTelemetry export.

Quick start

Install the plugin

openclaw plugins install clawhub:@openclaw/diagnostics-prometheus

Enable the plugin

Config

{
  plugins: {
    allow: ["diagnostics-prometheus"],
    entries: {
      "diagnostics-prometheus": { enabled: true },
    },
  },
  diagnostics: {
    enabled: true,
  },
}

CLI

openclaw plugins enable diagnostics-prometheus

Restart the Gateway

The HTTP route registers when the plugin starts, so reload after enabling.

Scrape the protected route

Pass the same gateway auth that your operator clients use:

curl -H "Authorization: Bearer $OPENCLAW_GATEWAY_TOKEN" \
  http://127.0.0.1:18789/api/diagnostics/prometheus

Wire Prometheus

# prometheus.yml
scrape_configs:
  - job_name: openclaw
    scrape_interval: 30s
    metrics_path: /api/diagnostics/prometheus
    authorization:
      credentials_file: /etc/prometheus/openclaw-gateway-token
    static_configs:
      - targets: ["openclaw-gateway:18789"]

Note

diagnostics.enabled is set to true by default; change it to false only in tightly constrained environments. When it is false, the plugin still registers the HTTP route, but no diagnostic events reach the exporter, so the response comes back empty.

Metrics exported

For metrics tracking model calls, observation_unit="request" records a single observable provider request. observation_unit="turn" tracks a synthetic Claude Code or Codex CLI agent turn, which may include multiple hidden provider requests. Keep these two series distinct when evaluating latency.

Label policy

Bounded, low-cardinality labels

Prometheus labels remain bounded and low in cardinality. The exporter does not expose raw diagnostic identifiers like runId, sessionKey, sessionId, callId, toolCallId, message IDs, chat IDs, or provider request IDs.

Label values are sanitized and must comply with OpenClaw's low-cardinality character rules. Values that violate the policy get replaced with unknown, other, or none, depending on the specific metric. Labels resembling scoped agent session keys are also substituted with unknown.

Series cap and overflow accounting

The exporter limits retained time series in memory to a total of 2048 series across counters, gauges, and histograms combined. Any new series beyond that limit are discarded, and openclaw_prometheus_series_dropped_total increases by one for each dropped series.

Monitor this counter as a clear indicator that an upstream attribute is leaking high-cardinality values. The exporter never raises the limit on its own; if the counter rises, address the root cause instead of turning off the cap.

What never appears in Prometheus output

• prompt text, response text, tool inputs, tool outputs, system prompts
• Talk transcripts, audio payloads, call ids, room ids, handoff tokens, turn ids, and raw session ids
• raw provider request IDs (only bounded hashes, where applicable, on spans, never on metrics)
• session keys and session IDs
• hostnames, file paths, secret values

PromQL recipes

# Tokens per minute, split by provider
sum by (provider) (rate(openclaw_model_tokens_total[1m]))

# Spend (USD) over the last hour, by model
sum by (model) (increase(openclaw_model_cost_usd_total[1h]))

# 95th percentile model run duration
histogram_quantile(
  0.95,
  sum by (le, provider, model)
    (rate(openclaw_run_duration_seconds_bucket[5m]))
)

# Queue wait time SLO (95p under 2s)
histogram_quantile(
  0.95,
  sum by (le, lane) (rate(openclaw_queue_lane_wait_seconds_bucket[5m]))
) < 2

# Skill usage, split by bounded source
sum by (skill, source) (increase(openclaw_skill_used_total[24h]))

# Dropped Prometheus series (cardinality alarm)
increase(openclaw_prometheus_series_dropped_total[15m]) > 0

Tip

For dashboards that span multiple providers, choose gen_ai_client_token_usage: it adheres to the OpenTelemetry GenAI semantic conventions and aligns with metrics from non-OpenClaw GenAI services.

Choosing between Prometheus and OpenTelemetry export

OpenClaw supports both surfaces independently. You can run either one, both, or neither.

diagnostics-prometheus

• Pull model: Prometheus scrapes /api/diagnostics/prometheus.
• No external collector is needed.
• Authentication uses standard Gateway auth.
• Surface provides metrics only (no traces or logs).
• Ideal for stacks already standardized on Prometheus and Grafana.

diagnostics-otel

• Push model: OpenClaw sends OTLP/HTTP to a collector or an OTLP-compatible backend.
• Surface includes metrics, traces, and logs.
• Bridges to Prometheus through an OpenTelemetry Collector (using the prometheus or prometheusremotewrite exporter) when both are required.
• See OpenTelemetry export for the complete catalog.

Troubleshooting

Empty response body

• Verify that diagnostics.enabled is not configured to false (it defaults to true).
• Ensure the plugin is enabled and loaded using openclaw plugins list --enabled.
• Generate some traffic; counters and histograms only produce lines after at least one event occurs.

401 / unauthorized

The endpoint requires the Gateway operator scope (auth: "gateway" with gatewayRuntimeScopeSurface: "trusted-operator"). Use the same token or password that Prometheus uses for any other Gateway operator route. No public unauthenticated mode exists.

openclaw_prometheus_series_dropped_total is climbing

A new attribute is exceeding the 2048-series limit. Check recent metrics for a label with unexpectedly high cardinality and resolve it at the source. The exporter deliberately drops new series rather than silently rewriting labels.

Prometheus shows stale series after a restart

The plugin stores state only in memory. After a Gateway restart, counters reset to zero and gauges restart at their next reported value. Use PromQL rate() and increase() to handle resets cleanly.

Related

• Diagnostics export, local diagnostics zip for support bundles
• Health and readiness, /healthz and /readyz probes
• Logging, file-based logging
• OpenTelemetry export, OTLP push for traces, metrics, and logs