briancaffey
hermes-otel
Export Hermes Agent LLM traces, tool calls, and API requests as OpenTelemetry spans to any OTLP-compatible backend.
Repository last updated Jul 11, 2026
Overview
hermes-otel is an OpenTelemetry plugin for Hermes Agent that automatically captures LLM tool calls, model invocations, and API requests as OTel spans. It exports these spans to any OTLP HTTP endpoint, enabling observability across a wide range of backends.
The plugin supports both single-backend configuration via environment variables and multi-backend fan-out via a config.yaml file. Each backend gets its own BatchSpanProcessor and, where supported, PeriodicExportingMetricReader, ensuring that a slow or unreachable collector does not block the agent's hot path.
Tested backends include Phoenix, Langfuse, LangSmith, SigNoz, Jaeger, Grafana Tempo, Grafana LGTM, Uptrace, OpenObserve, Honeycomb, and W&B Weave. Any OTLP HTTP endpoint should work. The plugin also provides a comprehensive test suite with unit, integration, E2E, and smoke tests.
Highlights
- Exports LLM tool calls and model invocations as OTel spans
- Supports OTLP HTTP endpoints for any compatible backend
- Multi-backend fan-out via config.yaml with parallel export
- Full-conversation capture with configurable history limit
- Comprehensive test suite: unit, integration, E2E, and smoke tests
Features
Multi-Backend Support
Export traces and metrics to one or more backends simultaneously via environment variables or a config.yaml file.
Parallel Export
Each backend gets its own BatchSpanProcessor and background worker, so a slow collector cannot block others.
Full-Conversation Capture
Optionally capture the entire message list the model saw, with a configurable character limit.
Test Suite Tiers
Unit tests run in under a second, integration tests use InMemorySpanExporter, E2E tests target real backends, and smoke tests exercise the full pipeline.
Installation
Commands are reproduced exactly as published by the project maintainers. Always check the upstream repository for the current instructions.
hermes plugins install briancaffey/hermes-otel# Install OTel runtime dependencies into the hermes-agent venv
~/git/hermes-agent/venv/bin/pip install \
opentelemetry-api \
opentelemetry-sdk \
opentelemetry-exporter-otlp-proto-http
# Optional: for LangSmith time-ordered run IDs
~/git/hermes-agent/venv/bin/pip install langsmith~/git/hermes-agent/venv/bin/pip install -e ~/.hermes/plugins/hermes_otelcd ~/.hermes/plugins/hermes_otel
# Unit + integration tests (no Docker needed, <1s)
uv run --extra dev pytest
# All E2E tests (requires Docker)
uv run --extra dev --extra e2e pytest -m e2e
# Phoenix E2E only (starts a single container)
uv run --extra dev --extra e2e pytest -m phoenix
# Langfuse E2E only (starts full stack via docker compose)
uv run --extra dev --extra e2e pytest -m langfuse
# Smoke tests — full pipeline: hermes API server -> plugin -> Langfuse
uv run --extra dev --extra e2e pytest -m smokeRequirements
- Hermes Agent installed and configured
- Python 3.8+
- opentelemetry-api, opentelemetry-sdk, opentelemetry-exporter-otlp-proto-http installed in the Hermes Agent venv
- Docker (optional, for E2E and smoke tests)
How it's built
The plugin is written in Python and lives in ~/.hermes/plugins/hermes_otel/. It hooks into Hermes Agent via callbacks defined in hooks.py, exporting spans for LLM tool calls, model invocations, and API requests. The OTel runtime dependencies (opentelemetry-api, opentelemetry-sdk, opentelemetry-exporter-otlp-proto-http) must be installed into the Hermes Agent virtual environment.
Configuration is handled either through environment variables (e.g., OTEL_PHOENIX_ENDPOINT) or a config.yaml file in the plugin root. The config.yaml.example file provides a fully annotated template. Secrets should be referenced via *_env keys rather than stored inline.
The repository includes a dashboard built with Preact and TypeScript under dashboard/, and Docker Compose files for various backends under docker-compose/. The test suite uses uv for dependency management and supports markers for unit, integration, E2E, and smoke tests.
Source
This summary was written for Neura Market from the project's own repository. Install commands, identifiers and licence details are reproduced unchanged. The briancaffey/hermes-otel repository is the authoritative source and the place to file issues or contribute.
Category: Observability & Telemetry · Licensed under Apache-2.0