Hermes Agent TTS and STT Integration Guide

# In ~/.hermes/config.yaml
tts:
  provider: "edge" # "edge" | "elevenlabs" | "openai" | "minimax" | "mistral" | "gemini" | "xai" | "neutts" | "kittentts" | "piper"
  speed: 1.0 # Global speed multiplier (provider-specific settings override this)
  edge:
    voice: "en-US-AriaNeural" # 322 voices, 74 languages
    speed: 1.0 # Converted to rate percentage (+/-%)
  elevenlabs:
    voice_id: "pNInz6obpgDQGcFmaJgB" # Adam
    model_id: "eleven_multilingual_v2"
  openai:
    model: "gpt-4o-mini-tts"
    voice: "alloy" # alloy, echo, fable, onyx, nova, shimmer
    base_url: "https://api.openai.com/v1" # Override for OpenAI-compatible TTS endpoints
    speed: 1.0 # 0.25 - 4.0
  minimax:
    model: "speech-02-hd" # speech-02-hd (default), speech-02-turbo
    voice_id: "English_Graceful_Lady" # See https://platform.minimax.io/faq/system-voice-id
    speed: 1 # 0.5 - 2.0
    vol: 1 # 0 - 10
    pitch: 0 # -12 - 12
  mistral:
    model: "voxtral-mini-tts-2603"
    voice_id: "c69964a6-ab8b-4f8a-9465-ec0925096ec8" # Paul - Neutral (default)
  gemini:
    model: "gemini-2.5-flash-preview-tts" # or gemini-3.1-flash-tts-preview
    voice: "Kore" # 30 prebuilt voices: Zephyr, Puck, Kore, Enceladus, Gacrux, etc.
    audio_tags: false # Enable hidden Gemini 3.1 TTS audio-tag insertion
    persona_prompt_file: "" # Optional Markdown/text file with Gemini voice direction
  xai:
    voice_id: "eve" # or a custom voice ID, see docs below
    language: "en" # ISO 639-1 code
    sample_rate: 24000 # 22050 / 24000 (default) / 44100 / 48000
    bit_rate: 128000 # MP3 bitrate; only applies when codec=mp3
    # base_url: "https://api.x.ai/v1" # Override via XAI_BASE_URL env var
  neutts:
    ref_audio: ''
    ref_text: ''
    model: neuphonic/neutts-air-q4-gguf
    device: cpu
  kittentts:
    model: KittenML/kitten-tts-nano-0.8-int8 # 25MB int8; also: kitten-tts-micro-0.8 (41MB), kitten-tts-mini-0.8 (80MB)
    voice: Jasper # Jasper, Bella, Luna, Bruno, Rosie, Hugo, Kiki, Leo
    speed: 1.0 # 0.5 - 2.0
    clean_text: true # Expand numbers, currencies, units
  piper:
    voice: en_US-lessac-medium # voice name (auto-downloaded) OR absolute path to .onnx
    # voices_dir: '' # default: ~/.hermes/cache/piper-voices/
    # use_cuda: false # requires onnxruntime-gpu
    # length_scale: 1.0 # 2.0 = twice as slow
    # noise_scale: 0.667
    # noise_w_scale: 0.8
    # volume: 1.0 # 0.5 = half as loud
    # normalize_audio: true

# Ubuntu/Debian
sudo apt install ffmpeg

# macOS
brew install ffmpeg

# Fedora
sudo dnf install ffmpeg

# In ~/.hermes/config.yaml
stt:
  provider: "local" # "local" | "groq" | "openai" | "mistral" | "xai"
  local:
    model: "base" # tiny, base, small, medium, large-v3
  openai:
    model: "whisper-1" # whisper-1, gpt-4o-mini-transcribe, gpt-4o-transcribe
  mistral:
    model: "voxtral-mini-latest" # voxtral-mini-latest, voxtral-mini-2602
  xai:
    model: "grok-stt" # xAI Grok STT

tts:
  openai:
    max_text_length: 8192 # raise or lower the provider cap

How It Works

TTS Architecture

Hermes Agent's TTS system is provider-agnostic. When the agent generates a text response, the text_to_speech tool routes the text to the configured provider. The provider returns audio bytes, which Hermes formats according to the target platform's requirements (Opus/OGG for Telegram voice bubbles, MP3 for WhatsApp attachments, etc.).

Speed control works hierarchically: the global tts.speed value applies to all providers by default. Each provider can override it with its own speed setting (e.g., tts.openai.speed: 1.5). Provider-specific speed takes precedence over the global value. The default is 1.0 (normal speed).

Custom Command Providers

If a TTS engine is not natively supported, you can wire it in as a command-type provider without writing any Python. Hermes writes the input text to a temp UTF-8 file, runs your shell command, and reads the audio file the command produced. Declare one or more providers under tts.providers.<name> and switch between them with tts.provider: <name>.

Placeholders

Your command template can reference these placeholders. Hermes substitutes them at render time and shell-quotes each value for the surrounding context (bare / single-quoted / double-quoted), so paths with spaces and other shell-sensitive characters are safe.

Placeholder	Meaning
{input_path}	Path to the temp UTF-8 text file Hermes wrote
{text_path}	Alias for {input_path}
{output_path}	Path the command must write audio to
{format}	mp3 / wav / ogg / flac
{voice}	tts.providers.<name>.voice, empty when unset
{model}	tts.providers.<name>.model
{speed}	Resolved speed multiplier (provider or global)

Use {{ and }} for literal braces.

Optional Keys

Key	Default	Meaning
timeout	120	Seconds; the process tree is killed on expiry (Unix killpg, Windows taskkill /T)
output_format	mp3	One of mp3 / wav / ogg / flac. Auto-inferred from the output extension if Hermes picks a path
voice_compatible	false	When true, Hermes converts MP3/WAV output to Opus/OGG via ffmpeg so Telegram renders a voice bubble
max_text_length	5000	Input is truncated to this length before rendering the command
voice / model	empty	Passed to the command as placeholder values only

Behavior Notes

• Built-in names always win. A tts.providers.openai entry never shadows the native OpenAI provider, so no user config can silently replace a built-in.
• Default delivery is a document. Command providers deliver as regular audio attachments on every platform. Opt in to voice-bubble delivery per-provider with voice_compatible: true.
• Command failures surface to the agent. Non-zero exit, empty output, or timeout all return an error with the command's stderr/stdout included so you can debug the provider from the conversation.
• type: command is the default when command: is set. Writing type: command explicitly is good practice but not required; an entry with a non-empty command string is treated as a command provider.
• {input_path} / {text_path} are interchangeable. Use whichever reads better in your command.

Security

Command-type providers run whatever shell command you configure, with your user's permissions. Hermes quotes placeholder values and enforces the configured timeout, but the command template itself is trusted local input. Treat it the same way you would a shell script on your PATH.

Python Plugin Providers

For TTS engines that cannot be expressed as a single shell command (Python SDKs without a CLI, streaming engines, voice-listing APIs, OAuth-refreshing auth), register a Python plugin via ctx.register_tts_provider(). The plugin coexists with the Custom command providers registry; pick the surface that fits your engine.

When to Pick Which

Your backend has…	Use
A single CLI reading text from a file/stdin and writing audio to a file/stdout	Command provider (no Python needed)
Two or three CLIs chained with shell pipes	Command provider
A Python SDK only, no CLI	Plugin
Streaming bytes you want to deliver chunked (mid-generation voice bubbles)	Plugin (override stream())
A voice-listing API used by hermes setup	Plugin (override list_voices())
OAuth refresh flow (not a static bearer token)	Plugin

Built-ins always win, and command providers win over a same-name plugin. So plugins are safe to register against any non-built-in name without worrying about shadowing your existing config.

Minimal Plugin Example

Drop this in ~/.hermes/plugins/my-tts/:

plugin.yaml:

name: my-tts
version: 0.1.0
description: "My custom Python TTS backend"

__init__.py:

from agent.tts_provider import TTSProvider

class MyTTSProvider(TTSProvider):
    @property
    def name(self) -> str:
        return "my-tts" # what tts.provider matches against

    @property
    def display_name(self) -> str:
        return "My Custom TTS"

    def is_available(self) -> bool:
        # Return False when credentials/deps are missing, picker skips
        # this row but the dispatcher still routes here on explicit config.
        import os
        return bool(os.environ.get("MY_TTS_API_KEY"))

    def synthesize(self, text, output_path, *, voice=None, model=None, speed=None, format="mp3", **extra) -> str:
        # Write audio bytes to output_path, return the path.
        # Raise on failure, the dispatcher converts exceptions to a
        # standard error envelope.
        import my_tts_sdk
        client = my_tts_sdk.Client()
        audio_bytes = client.synthesize(text=text, voice=voice or "default")
        with open(output_path, "wb") as f:
            f.write(audio_bytes)
        return output_path

def register(ctx):
    ctx.register_tts_provider(MyTTSProvider())

Enable it (hermes plugins enable my-tts), point tts.provider at it (tts.provider: my-tts in config.yaml), and the text_to_speech tool will route through your plugin.

Optional Hooks

Override these on your provider class for richer integration:

• list_voices() → list of {id, display, language, gender, preview_url} dicts shown in hermes tools.
• list_models() → list of {id, display, languages, max_text_length} dicts.
• get_setup_schema() → return {name, badge, tag, env_vars: [{key, prompt, url}]} to power the picker row in hermes tools / hermes setup. Without this, the plugin still works but its row in the picker is minimal.
• stream(text, *, voice, model, format, **extra) → iterator yielding audio bytes for streaming delivery (default raises NotImplementedError).
• voice_compatible property → set True if your output is Opus-compatible and the gateway should deliver it as a voice bubble (default False = regular audio attachment).

See agent/tts_provider.py for the full ABC including docstrings.

STT Architecture

Voice messages sent on Telegram, Discord, WhatsApp, Slack, or Signal are automatically transcribed and injected as text into the conversation. The agent sees the transcript as normal text.

Fallback Behavior

If your configured provider is not available, Hermes automatically falls back:

• Local faster-whisper unavailable → Tries a local whisper CLI or HERMES_LOCAL_STT_COMMAND before cloud providers.
• Groq key not set → Falls back to local transcription, then OpenAI.
• OpenAI key not set → Falls back to local transcription, then Groq.
• Mistral key/SDK not set → Skipped in auto-detect; falls through to next available provider.
• Nothing available → Voice messages pass through with an accurate note to the user.

Auto-detection order puts xAI after Groq. Explicitly set stt.provider: xai to force it.

STT Custom Command Providers

If the STT engine you want is not natively supported (Doubao ASR, NVIDIA Parakeet, a whisper.cpp build, an open-source SenseVoice CLI, anything else that exposes a shell command), wire it in as a command-type provider without writing any Python. Hermes runs your shell command against the audio file and reads back the transcript.

Declare one or more providers under stt.providers.<name> and switch between them with stt.provider: <name>, same shape as the TTS command-provider registry, adapted for the input=audio → output=transcript direction.

stt:
  provider: parakeet # pick any name under stt.providers
  providers:
    parakeet:
      type: command
      command: "parakeet-asr --model nvidia/parakeet-tdt-0.6b-v2 --in {input_path} --out {output_path}"
      format: txt
      language: en
      timeout: 300
    whispercpp:
      type: command
      command: "whisper-cli -m ~/models/ggml-large-v3.bin -f {input_path} -otxt -of {output_dir}/transcript"
      format: txt
    sensevoice:
      type: command
      command: "sensevoice-cli {input_path} --json | tee {output_path}"
      format: json

This complements the legacy HERMES_LOCAL_STT_COMMAND escape hatch. That env var still works untouched via the built-in local_command path. Use stt.providers.<name> when you want multiple shell-driven STT engines, a name you can pick via stt.provider, or anything that needs per-provider language / model / timeout.

STT Placeholders

Placeholder	Meaning
{input_path}	Absolute path to the input audio file (original location, read-only)
{output_path}	Absolute path the command should write the transcript to
{output_dir}	Parent directory of {output_path} (handy for whisper-style tools)
{format}	Configured output format: txt / json / srt / vtt
{language}	Configured language code (defaults to en)
{model}	stt.providers.<name>.model, empty when unset

Use {{ and }} for literal braces (handy when embedding JSON snippets in the command).

How the Transcript Is Read Back

After your command exits successfully:

• If {output_path} exists and is non-empty → Hermes reads it as UTF-8 text.
• Otherwise, if the command wrote to stdout → Hermes uses that.
• Otherwise → error: "Command STT provider wrote no output file and produced no stdout".

This lets you use the registry for tools that write to a file, tools that write to stdout, or tools that do both.

Requirements and Limitations

TTS Requirements

• Edge TTS: No API key. Works on any platform with internet access.
• ElevenLabs: Requires ELEVENLABS_API_KEY. Paid service.
• OpenAI TTS: Requires VOICE_TOOLS_OPENAI_KEY. Paid service. Also available through Nous Portal subscription without a separate key.
• MiniMax TTS: Requires MINIMAX_API_KEY. Paid service.
• Mistral TTS: Requires MISTRAL_API_KEY. Paid service.
• Google Gemini TTS: Requires GEMINI_API_KEY. Free tier available.
• xAI TTS: Requires XAI_API_KEY. Paid service.
• NeuTTS: No API key. Local. Requires pip install neutts.
• KittenTTS: No API key. Local. Requires pip install kitten-tts.
• Piper: No API key. Local. Requires pip install piper-tts. Runs entirely on CPU. Supports 44 languages with pre-trained voices.

STT Requirements

• Local Whisper: No API key. Requires faster-whisper installed. Runs on CPU by default, GPU if available.
• Groq Whisper API: Requires GROQ_API_KEY. Free tier available.
• OpenAI Whisper API: Requires VOICE_TOOLS_OPENAI_KEY or OPENAI_API_KEY. Paid service.
• Mistral Voxtral Transcribe: Requires MISTRAL_API_KEY. Supports 13 languages, speaker diarization, and word-level timestamps. Requires cd ~/.hermes/hermes-agent && uv pip install -e ".[mistral]".
• xAI Grok STT: Requires XAI_API_KEY. Posts to https://api.x.ai/v1/stt as multipart/form-data.

Known Limitations

• Telegram Voice Bubbles: Require Opus/OGG format. Providers that output MP3, WAV, or PCM need ffmpeg installed for conversion. Without ffmpeg, audio is sent as a regular file attachment.
• Piper Advanced Knobs: length_scale, noise_scale, noise_w_scale, volume, normalize_audio, and use_cuda correspond 1:1 to Piper's SynthesisConfig. They are ignored on older piper-tts versions.
• Custom Command Providers: Built-in names always win. A tts.providers.openai entry never shadows the native OpenAI provider. Default delivery is a document; opt in to voice-bubble delivery with voice_compatible: true.
• Input Length Override: Only positive integers are honored. Zero, negative, non-numeric, or boolean values fall through to the provider default.
• STT Auto-Detection Order: xAI is placed after Groq. Explicitly set stt.provider: xai to force it.
• Mistral STT: If the key or SDK is not set, Mistral is skipped in auto-detect and falls through to the next available provider.

Troubleshooting

TTS Issues

Problem: No audio output or incorrect provider used.

Fix: Verify tts.provider is set correctly in ~/.hermes/config.yaml. Check that the required API key environment variable is set. For local providers, confirm the package is installed (e.g., pip list | grep piper-tts).

Problem: Telegram voice bubble not appearing; audio sent as file attachment instead.

Fix: Install ffmpeg (see installation commands above). If you do not want to install ffmpeg, switch to OpenAI, ElevenLabs, or Mistral provider, which output Opus natively.

Problem: TTS request fails with a length error.

Fix: The provider has a character cap. Hermes truncates text automatically, but if you need longer input, override the cap with max_text_length: under the provider section. Only positive integers are honored.

Problem: Custom command provider returns an error.

Fix: Check the command's stderr/stdout, which is included in the error message. Verify the command template uses correct placeholders. Ensure the command writes audio to {output_path}. Increase timeout if the command takes longer than 120 seconds.

Problem: Piper voice not found.

Fix: On the first TTS call for a voice that is not cached locally, Hermes runs python -m piper.download_voices and downloads the model (~20-90MB depending on quality tier) into ~/.hermes/cache/piper-voices/. Ensure internet access for the download. Subsequent calls reuse the cached model.

Problem: Gemini audio tags not working.

Fix: Ensure tts.gemini.audio_tags is set to true. The rewrite uses auxiliary.tts_audio_tags and defaults to your main chat model. Override that auxiliary task if you want tag insertion handled by a cheaper or faster model.

STT Issues

Problem: Voice messages not being transcribed.

Fix: Check that stt.provider is set correctly. Verify the required API key is set for cloud providers. For local Whisper, confirm faster-whisper is installed. If nothing is available, voice messages pass through with an accurate note to the user.

Problem: Custom STT command provider produces no output.

Fix: Ensure the command writes a .txt file to {output_path} or produces stdout. If the command writes to a different location, adjust the command template. Check the timeout; increase if the command takes longer than the default.

Problem: Mistral STT not working.

Fix: Confirm MISTRAL_API_KEY is set. Install the Mistral extras with cd ~/.hermes/hermes-agent && uv pip install -e ".[mistral]". If the key or SDK is not set, Mistral is skipped in auto-detect.

Problem: xAI Grok STT not being used despite having the key.

Fix: Auto-detection order puts xAI after Groq. Explicitly set stt.provider: xai to force it.