Hermes Agent TTS and STT Integration Guide

{ "title": "Hermes Agent Voice Integration Reference", "body": "## Overview\n\nHermes Agent supports voice capabilities across Telegram, Discord, WhatsApp, and CLI. You can enable text-to-speech (TTS) for voice replies, configure speech-to-text (STT) to automatically transcribe incoming voice messages, and choose from multiple providers based on cost, quality, or privacy requirements. The system supports cloud and local providers, custom command-based engines, and Python plugins for full SDK integration.\n\n## Prerequisites\n\nBefore configuring voice features, ensure the following are in place:\n\n- Hermes Agent installed and configured.\n- For cloud TTS/STT providers: corresponding API keys set as environment variables.\n- For Nous Portal subscribers: paid subscription and logged in via hermes setup --portal.\n- For Piper TTS: install piper-tts via hermes tools or pip install piper-tts.\n- For local Whisper STT: install faster-whisper.\n- For Mistral STT: install with cd ~/.hermes/hermes-agent && uv pip install -e \".[mistral]\".\n- For Telegram voice bubbles with non-native Opus providers: install ffmpeg.\n- For custom command providers: the CLI tool must be installed and on PATH.\n- For Python plugins: plugin directory in ~/.hermes/plugins/<name>/ with plugin.yaml and __init__.py.\n- For xAI custom voice: create in xAI Console.\n- For Gemini persona prompts: Markdown/text file with voice directions.\n- For Doubao TTS/STT: install doubao-speech PyPI package and set VOLCENGINE_APP_ID, VOLCENGINE_ACCESS_TOKEN env vars.\n- For STT local_command: set HERMES_LOCAL_STT_COMMAND env var (optional).\n\n## TTS Providers\n\nHermes Agent supports 11 TTS providers. The default provider is Edge TTS, which is free and requires no API key.\n\n| Provider | Type | Default | Notes |\n|----------|------|---------|-------|\n| Edge TTS | Cloud (free) | Yes | No API key required |\n| ElevenLabs | Cloud (paid) | No | Requires ELEVENLABS_API_KEY |\n| OpenAI | Cloud (paid) | No | Requires VOICE_TOOLS_OPENAI_KEY |\n| MiniMax | Cloud (paid) | No | Requires MINIMAX_API_KEY or MINIMAX_CN_API_KEY |\n| Mistral | Cloud (paid) | No | Requires MISTRAL_API_KEY |\n| Google Gemini | Cloud (paid) | No | Requires GEMINI_API_KEY |\n| xAI | Cloud (paid) | No | Requires XAI_API_KEY |\n| DeepInfra | Cloud (paid) | No | Requires DEEPINFRA_API_KEY |\n| NeuTTS | Cloud (paid) | No | Requires NEUTTS_API_KEY |\n| KittenTTS | Cloud (paid) | No | Requires KITTENTTS_API_KEY |\n| Piper | Local (free) | No | CPU-only, no API key |\n\n## STT Providers\n\nHermes Agent supports 8 STT providers. The default provider is Local Whisper, which is free and runs locally.\n\n| Provider | Type | Default | Notes |\n|----------|------|---------|-------|\n| Local Whisper | Local (free) | Yes | Requires faster-whisper |\n| Groq | Cloud (paid) | No | Requires GROQ_API_KEY |\n| OpenAI | Cloud (paid) | No | Requires VOICE_TOOLS_OPENAI_KEY |\n| Mistral | Cloud (paid) | No | Requires MISTRAL_API_KEY |\n| xAI | Cloud (paid) | No | Requires XAI_API_KEY |\n| ElevenLabs | Cloud (paid) | No | Requires ELEVENLABS_API_KEY |\n| DeepInfra | Cloud (paid) | No | Requires DEEPINFRA_API_KEY |\n| local_command | Local (custom) | No | Uses HERMES_LOCAL_STT_COMMAND |\n\n## Configuring TTS\n\n### Configure TTS Provider\n\n1. Edit ~/.hermes/config.yaml.\n2. Set tts.provider to one of: edge, elevenlabs, openai, minimax, mistral, gemini, xai, deepinfra, neutts, kittentts, piper.\n3. Optionally set tts.speed (global multiplier, default 1.0).\n4. Add provider-specific section (e.g., tts.edge.voice, tts.openai.model).\n5. For paid providers, set the corresponding API key environment variable (e.g., ELEVENLABS_API_KEY, VOICE_TOOLS_OPENAI_KEY).\n6. For Nous Portal subscribers: run hermes setup --portal to log in and enable gateway tools; existing installs use hermes model or hermes tools to select Nous Subscription for TTS.\n7. For Piper: install via hermes tools or pip install piper-tts; set tts.piper.voice to a voice name or .onnx path.\n8. For custom command provider: add under tts.providers.<name> with type: command, command template, output_format, and optional keys (timeout, voice_compatible, max_text_length).\n9. For Python plugin: create plugin in ~/.hermes/plugins/<name>/ with plugin.yaml and __init__.py implementing TTSProvider; enable with hermes plugins enable <name>.\n10. Switch between providers by changing tts.provider.\n\n### TTS Parameters\n\n| Parameter | Meaning | Required |\n|-----------|---------|----------|\n| tts.provider | Selects the TTS engine: edge, elevenlabs, openai, minimax, mistral, gemini, xai, deepinfra, neutts, kittentts, piper, or a custom command/plugin name | Yes |\n| tts.speed | Global speed multiplier (default 1.0); overridable per provider | No |\n| tts.<provider>.voice / voice_id | Voice identifier for the provider (e.g., en-US-AriaNeural for Edge, Adam for ElevenLabs, alloy for OpenAI) | Depends on provider |\n| tts.<provider>.model / model_id | Model identifier (e.g., gpt-4o-mini-tts, eleven_multilingual_v2, speech-02-hd) | Depends on provider |\n| tts.<provider>.speed | Provider-specific speed override; takes precedence over global tts.speed | No |\n| tts.<provider>.language | Language code for OpenAI-compatible endpoints (sent as lang_code); leave unset for official OpenAI API | No |\n| tts.<provider>.max_text_length | Override per-provider input character cap; only positive integers honored | No |\n| tts.minimax.region | 'global' or 'cn'; selects endpoint and credential pair | No |\n| tts.gemini.audio_tags | Enable hidden rewrite to insert expressive audio tags for Gemini 3.1 Flash TTS | No |\n| tts.gemini.persona_prompt_file | Path to Markdown/text file with voice persona directions | No |\n| tts.xai.auto_speech_tags | Enable hidden rewrite to insert expressive audio tags for xAI Grok TTS | No |\n| tts.xai.voice_id | Voice ID (built-in like 'eve' or custom cloning ID) | Yes for xAI |\n| tts.xai.language | BCP-47 code or 'auto' for detection | No |\n| tts.xai.speed | Playback speed 0.7–1.5 (default 1.0) | No |\n| tts.xai.optimize_streaming_latency | 0–2, trades quality for lower latency (default 0) | No |\n| tts.xai.sample_rate | 22050, 24000 (default), 44100, 48000 | No |\n| tts.xai.bit_rate | MP3 bitrate; applies only when codec=mp3 | No |\n| tts.xai.text_normalization | Normalize numbers/abbreviations/symbols to spoken form | No |\n| tts.piper.voice | Voice name (auto-downloaded) or absolute path to .onnx file | Yes for Piper |\n| tts.piper.length_scale / noise_scale / noise_w_scale / volume / normalize_audio / use_cuda | Advanced Piper SynthesisConfig knobs; ignored on older piper-tts | No |\n\n### TTS Command Provider Parameters\n\n| Parameter | Meaning | Required |\n|-----------|---------|----------|\n| tts.providers.<name>.command | Shell command template with placeholders for custom TTS provider | Yes for command provider |\n| tts.providers.<name>.output_format | Audio format: mp3 (default), wav, ogg, flac, m4a, aac, amr, opus | No |\n| tts.providers.<name>.timeout | Idle seconds before process tree killed (default 120) | No |\n| tts.providers.<name>.voice_compatible | When true, convert to Opus/OGG for Telegram voice bubbles | No |\n| tts.providers.<name>.max_text_length | Input truncation length (default 5000) | No |\n| tts.providers.<name>.env_passthrough | List of env var names to keep in subprocess environment | No |\n\n### Input Length Limits\n\nOverride input length limits by adding max_text_length under the provider section (e.g., tts.openai.max_text_length: 8192). Only positive integers are honored; zero, negative, non-numeric, or boolean values fall through to the provider default.\n\nProvider defaults:\n- Edge TTS: 5000\n- OpenAI: 4096\n- xAI: 15000\n- MiniMax: 10000\n- Mistral: 4000\n- Gemini: 32000\n- ElevenLabs: model-aware\n- NeuTTS/KittenTTS: 2000\n- Piper: 5000\n\nElevenLabs caps by model:\n- eleven_flash_v2_5: 40000\n- eleven_flash_v2: 30000\n- eleven_multilingual_v2/v1/english_sts_v2/v1: 10000\n- eleven_v3/ttv_v3: 5000\n- Unknown model: 10000\n\n## Configuring STT\n\n### Configure STT Provider\n\n1. Edit ~/.hermes/config.yaml.\n2. Set stt.provider to one of: local, groq, openai, mistral, xai, elevenlabs, deepinfra, local_command.\n3. Optionally set stt.language (global hint, default 'en'; set to '' for auto-detect).\n4. Add provider-specific section (e.g., stt.local.model, stt.groq.language).\n5. For local Whisper: ensure faster-whisper is installed; model sizes: tiny, base (default), small, medium, large-v3.\n6. For cloud providers: set corresponding API key (e.g., GROQ_API_KEY, VOICE_TOOLS_OPENAI_KEY, MISTRAL_API_KEY, XAI_API_KEY).\n7. For Mistral: install with cd ~/.hermes/hermes-agent && uv pip install -e \".[mistral]\".\n8. For custom local CLI: set HERMES_LOCAL_STT_COMMAND env var with placeholders {input_path}, {output_dir}, {language}, {model}.\n9. For custom command provider: add under stt.providers.<name> with type: command, command template, format, and optional keys (timeout, language, model).\n10. For Python plugin: create plugin in ~/.hermes/plugins/<name>/ with plugin.yaml and __init__.py implementing TranscriptionProvider; enable with hermes plugins enable <name>.\n11. Switch between providers by changing stt.provider.\n\n### STT Parameters\n\n| Parameter | Meaning | Required |\n|-----------|---------|----------|\n| stt.provider | Selects the STT engine: local, groq, openai, mistral, xai, elevenlabs, deepinfra, local_command, or custom command/plugin name | Yes |\n| stt.language | Global language hint for all STT providers; set '' for auto-detect | No |\n| stt.local.model | Whisper model size: tiny, base (default), small, medium, large-v3 | No |\n| stt.local.language | Optional ISO-639-1 hint; blank = use HERMES_LOCAL_STT_LANGUAGE if set, else auto-detect | No |\n| stt.groq.language | Optional ISO-639-1 hint; blank = use HERMES_LOCAL_STT_LANGUAGE if set, else auto-detect | No |\n| stt.openai.model | Model: whisper-1, gpt-4o-mini-transcribe, gpt-4o-transcribe, gpt-transcribe | No |\n| stt.mistral.model | Model: voxtral-mini-latest, voxtral-mini-2602 | No |\n| stt.xai.model | Model: grok-stt | No |\n| stt.xai.language | Optional ISO-639-1 hint; blank = use HERMES_LOCAL_STT_LANGUAGE if set, else 'en' | No |\n\n### STT Command Provider Parameters\n\n| Parameter | Meaning | Required |\n|-----------|---------|----------|\n| stt.providers.<name>.command | Shell command template with placeholders for custom STT provider | Yes for command provider |\n| stt.providers.<name>.format | Output format: txt (default), json, srt, vtt | No |\n| stt.providers.<name>.timeout | Seconds before process tree killed (default 300) | No |\n| stt.providers.<name>.language | Language code forwarded to {language} placeholder; defaults to stt.language then 'en' | No |\n| stt.providers.<name>.model | Model identifier forwarded to {model} placeholder | No |\n\n### Legacy STT Environment Variables\n\n| Variable | Meaning |\n|----------|---------|\n| HERMES_LOCAL_STT_COMMAND | Environment variable for legacy local STT command template (argv-tokenized, no shell) |\n| HERMES_LOCAL_STT_LANGUAGE | Environment variable for local STT language hint |\n\n## Telegram Voice Bubbles\n\nTelegram voice bubbles require specific handling depending on the TTS provider.\n\n### Set Up Telegram Voice Bubbles\n\n1. Check if your TTS provider produces Opus natively (OpenAI, ElevenLabs, Mistral) – no extra setup.\n2. For other providers (Edge TTS, MiniMax, Gemini, xAI, NeuTTS, KittenTTS, Piper): install ffmpeg.\n3. Install ffmpeg:\n - Ubuntu/Debian: sudo apt install ffmpeg\n - macOS: brew install ffmpeg\n - Fedora: sudo dnf install ffmpeg\n4. Without ffmpeg, audio is sent as regular file (rectangular player, not voice bubble).\n5. Alternatively, switch to a native Opus provider to avoid ffmpeg dependency.\n\n## Gemini Persona Prompts\n\nGemini supports expressive TTS with persona prompts that define the voice character.\n\n### Use Gemini Persona Prompts\n\n1. Set tts.provider: gemini.\n2. Set tts.gemini.persona_prompt_file to a local Markdown or text file describing the voice persona.\n3. File can include sections: AUDIO PROFILE, SCENE, DIRECTOR'S NOTES, SAMPLE CONTEXT, TRANSCRIPT.\n4. If file contains {transcript} or {{ transcript }}, Hermes replaces with live TTS text; otherwise appends TRANSCRIPT section automatically.\n5. Persona prompt stays local, not shown in chat reply.\n\n## Audio Tags (Gemini, xAI)\n\nAudio tags add expressive markers like [whispers] or [excitedly] to TTS output.\n\n### Enable Audio Tags\n\n- For Gemini: set tts.gemini.audio_tags: true.\n- For xAI: set tts.xai.auto_speech_tags: true.\n\nHermes runs a hidden rewrite pass to insert square-bracket tags into the TTS script only; the visible chat reply remains unchanged. The rewrite uses the auxiliary.tts_audio_tags model (defaults to main chat model); override for a cheaper or faster model.\n\n## xAI Custom Voice Cloning\n\n### Set Up xAI Custom Voice Cloning\n\n1. Create a custom voice in xAI Console.\n2. Set tts.xai.voice_id to the resulting custom voice ID (e.g., 'nlbqfwie').\n3. See xAI Custom Voices docs for recording, formats, limits.\n\n## MiniMax TTS Region\n\nMiniMax offers two regional endpoints with separate credentials.\n\n### Configure MiniMax TTS Region\n\n1. Set tts.minimax.region to 'global' (uses api.minimax.io with MINIMAX_API_KEY) or 'cn' (uses api.minimaxi.com with MINIMAX_CN_API_KEY).\n2. If region omitted: MINIMAX_API_KEY takes precedence for backward compatibility; if only MINIMAX_CN_API_KEY set, Hermes selects cn.\n3. Explicit region must have matching credential; Hermes never borrows other region's key.\n4. base_url override does not change credential; override pointing at other region's official endpoint is rejected.\n\n## Piper Local Neural TTS\n\nPiper provides local neural TTS supporting 44 languages, CPU-only, with no API key required.\n\n### Configure Piper\n\n1. Set tts.provider: piper.\n2. Set tts.piper.voice to a voice name (auto-downloaded) or absolute path to .onnx file.\n3. First TTS call downloads the model (~20-90MB) to ~/.hermes/cache/piper-voices/.\n4. Advanced knobs (length_scale, noise_scale, noise_w_scale, volume, normalize_audio, use_cuda) are ignored on older piper-tts versions.\n\n## Custom Command Provider for TTS\n\nWire any CLI tool without writing Python.\n\n### Use Custom Command Provider for TTS\n\n1. Add under tts.providers.<name> with type: command (or omit if command: is set).\n2. Set command template with placeholders: {input_path}, {text_path}, {output_path}, {format}, {voice}, {model}, {speed}.\n3. Set output_format (mp3 default, wav, ogg, flac, m4a, aac, amr, opus).\n4. Optional keys: timeout (120s default), voice_compatible (false), max_text_length (5000), voice, model.\n5. For API keys in command: list env var names under env_passthrough.\n6. Built-in names always win; command provider cannot shadow native provider.\n7. Default delivery is document; set voice_compatible: true for voice bubbles.\n8. Command failures surface error with stderr/stdout.\n\n## Custom Command Provider for STT\n\nWire any CLI tool for transcription without writing Python.\n\n### Use Custom Command Provider for STT\n\n1. Add under stt.providers.<name> with type: command.\n2. Set command template with placeholders: {input_path}, {output_path}, {output_dir}, {format}, {language}, {model}.\n3. Set format (txt default, json, srt, vtt).\n4. Optional keys: timeout (300s default), language (en default), model.\n5. Built-in names always win.\n6. Transcript read from {output_path} if exists and non-empty; else from stdout; else error.\n7. For format json/srt/vtt, raw file content returned as transcript field; extract .text downstream if needed.\n8. Process-tree cleanup on timeout.\n9. Shell-quoting automatic; don't pre-quote placeholders.\n\n## Python Plugin for TTS\n\nFull SDK integration with optional hooks.\n\n### Use Python Plugin for TTS\n\n1. Create ~/.hermes/plugins/<name>/ with plugin.yaml (name, version, description) and __init__.py.\n2. In __init__.py: import TTSProvider from agent.tts_provider; create class inheriting TTSProvider; implement name, display_name, is_available, synthesize.\n3. synthesize signature: (self, text, output_path, *, voice=None, model=None, speed=None, format='mp3', **extra) -> str.\n4. Write audio bytes to output_path; raise on failure.\n5. Implement register(ctx) function calling ctx.register_tts_provider(MyTTSProvider()).\n6. Optional hooks: list_voices, list_models, get_setup_schema, stream, voice_compatible property.\n7. Enable plugin: hermes plugins enable <name>.\n8. Set tts.provider: <name> in config.yaml.\n\n## Python Plugin for STT\n\nFull SDK integration for transcription.\n\n### Use Python Plugin for STT\n\n1. Create ~/.hermes/plugins/<name>/ with plugin.yaml and __init__.py.\n2. In __init__.py: import TranscriptionProvider from agent.transcription_provider; create class inheriting TranscriptionProvider; implement name, display_name, is_available, transcribe.\n3. transcribe signature: (self, file_path, *, model=None, language=None, **extra) -> dict with keys success, transcript, provider, error.\n4. Do NOT raise exceptions; return error envelope.\n5. Implement register(ctx) function calling ctx.register_transcription_provider(MySTTProvider()).\n6. Optional hooks: list_models, default_model, get_setup_schema.\n7. Enable plugin: hermes plugins enable <name>.\n8. Set stt.provider: <name> in config.yaml.\n9. Plugin reads per-provider config from stt.<provider> in config.yaml (model, language, etc.).\n\n## STT Fallback Chain\n\nWhen the configured STT provider is unavailable, Hermes attempts an automatic fallback chain:\n\n- Local faster-whisper unavailable → local whisper CLI/HERMES_LOCAL_STT_COMMAND → cloud.\n- Groq key missing → local then OpenAI.\n- OpenAI key missing → local then Groq.\n- Mistral key/SDK missing → skipped in auto-detect.\n- Nothing available → voice messages pass through with note.\n\n## STT Resolution Order\n\nWhen resolving an STT provider, Hermes follows this order:\n\n1. Built-in name → built-in dispatch.\n2. Matches stt.providers.<name> with command → command provider.\n3. Matches plugin → plugin dispatch (if is_available false, surfaces plugin error).\n4. No match → 'No STT provider available'.\n\n## Constraints and Caveats\n\n- Built-in provider names always win over command providers or plugins of the same name.\n- Command providers for TTS default to document delivery; set voice_compatible: true for voice bubbles.\n- Command providers for STT: transcript read from {output_path} first, then stdout; else error.\n- STT command provider format json/srt/vtt returns raw file content; extract .text downstream.\n- MiniMax region selection: explicit region requires matching credential; Hermes never borrows other region's key; base_url override pointing at other region's official endpoint is rejected.\n- MiniMax region omitted: MINIMAX_API_KEY takes precedence; if only MINIMAX_CN_API_KEY set, Hermes selects cn.\n- Gemini persona prompt file stays local; not shown in chat reply.\n- Audio tags rewrite only affects TTS script; visible chat reply unchanged.\n- Input length limits: only positive integers honored for max_text_length; zero/negative/non-numeric/boolean fall through to provider default.\n- ElevenLabs cap depends on model_id; unknown model falls back to 10000.\n- Telegram voice bubbles: OpenAI, ElevenLabs, Mistral produce Opus natively; others require ffmpeg.\n- Without ffmpeg, audio sent as regular file (rectangular player, not voice bubble).\n- xAI custom voice: create in xAI Console; see docs for recording/format/limits.\n- Piper: first TTS call downloads model (~20-90MB) to ~/.hermes/cache/piper-voices/.\n- Piper advanced knobs ignored on older piper-tts versions.\n- Command provider environment: Hermes secrets scrubbed; PATH, HOME, locale kept; use env_passthrough for API keys.\n- STT fallback: local faster-whisper unavailable → local whisper CLI/HERMES_LOCAL_STT_COMMAND → cloud; Groq key missing → local then OpenAI; OpenAI key missing → local then Groq; Mistral key/SDK missing → skipped in auto-detect; nothing available → voice messages pass through with note.\n- STT resolution order: built-in name → built-in dispatch; matches stt.providers.<name> with command → command provider; matches plugin → plugin dispatch (if is_available false, surfaces plugin error); no match → 'No STT provider available'.\n- Plugin STT: transcribe() must return dict with success, transcript, provider, error; do NOT raise exceptions.\n- Plugin TTS: synthesize() must write audio to output_path and return path; raise on failure.\n- Command provider security: runs with user permissions; treat template as trusted local input.\n- HERMES_LOCAL_STT_COMMAND: tokenized into argv, no shell interpretation; use explicit sh -c for pipes/redirects.\n- On Windows, use explicit cmd /c or PowerShell wrapper for shell features in STT command.\n- Subprocess environment scrubbing applies to both TTS and STT command providers.\n- Process-tree cleanup on timeout for both TTS and STT command providers (Unix killpg, Windows taskkill /T).\n- Placeholder shell-quoting automatic; don't pre-quote values.\n- Use {{ and }} for literal braces in command templates.\n\n## Failure Modes\n\n- TTS command provider: non-zero exit, empty output, or timeout returns error with stderr/stdout.\n- STT command provider: no output file and no stdout → error 'Command STT provider wrote no output file and produced no stdout'.\n- STT plugin: is_available() returns False → surfaces unavailability error envelope identifying the plugin.\n- STT no match: 'No STT provider available' error.\n- MiniMax region mismatch: explicit region without matching credential → Hermes rejects; base_url override pointing at other region's official endpoint rejected.\n- Input length exceeded: Hermes truncates before calling provider; broken max_text_length config (zero/negative/non-numeric) falls through to provider default.\n- ffmpeg missing for Telegram voice bubbles: audio sent as regular file, not voice bubble.\n- Piper advanced knobs ignored on older piper-tts versions.\n- Command provider environment: missing env_passthrough for API keys → command may fail.\n- STT fallback exhaustion: nothing available → voice messages pass through with note to user.\n\n## Examples\n\n### Doubao TTS (Chinese seed-tts-2.0)\n\nInstall doubao-speech, set VOLCENGINE_APP_ID and VOLCENGINE_ACCESS_TOKEN env vars, then configure as a command provider:\n\nyaml\ntts:\n providers:\n doubao:\n command: doubao-speech say --text-file {input_path} --out {output_path}\n output_format: mp3\n max_text_length: 1024\n timeout: 30\n\n\n### Doubao/Volcengine ASR\n\nInstall doubao-speech, set env vars, then set HERMES_LOCAL_STT_COMMAND to doubao-speech transcribe {input_path} --out {output_dir}/transcript.txt and set stt.provider: local_command.\n\n### xAI Custom Voice\n\nCreate a custom voice in xAI Console, then set tts.xai.voice_id to the custom ID.\n\n### Gemini Persona Prompt\n\nSet tts.gemini.persona_prompt_file to ~/.hermes/tts/butler-voice.md with voice directions.\n\n### Audio Tags\n\nSet tts.gemini.audio_tags: true or tts.xai.auto_speech_tags: true for expressive delivery.\n\n### Piper Local TTS\n\nSet tts.provider: piper and tts.piper.voice: en_US-lessac-medium. The first call downloads the model.\n\n### Custom Command TTS Provider\n\nyaml\ntts:\n providers:\n voxcpm:\n command: voxcpm --ref ~/voice.wav --text-file {input_path} --out {output_path}\n output_format: mp3\n timeout: 180\n voice_compatible: true\n\n\n### Custom Command STT Provider\n\nyaml\nstt:\n providers:\n parakeet:\n command: parakeet-asr --model nvidia/parakeet-tdt-0.6b-v2 --in {input_path} --out {output_path}\n format: txt\n language: en\n timeout: 300\n\n\n### Minimal TTS Plugin\n\nCreate ~/.hermes/plugins/my-tts/ with plugin.yaml and __init__.py implementing TTSProvider with a synthesize method.\n\n### Minimal STT Plugin\n\nCreate ~/.hermes/plugins/my-stt/ with plugin.yaml and __init__.py implementing TranscriptionProvider with a transcribe method returning an error envelope.\n\n## Speed Control\n\nThe global speed multiplier tts.speed (default 1.0) can be overridden per provider with tts.<provider>.speed. This allows fine-tuning playback speed for individual providers without affecting others.\n\n## Language Hints\n\nLanguage hints can be set globally with stt.language (default 'en'; set to '' for auto-detect) or per provider (e.g., stt.groq.language, stt.xai.language). For OpenAI-compatible TTS endpoints, set tts.<provider>.language as lang_code; leave unset for the official OpenAI API.\n\n## Delivery Methods\n\nTTS audio is delivered as voice bubbles on Telegram and Discord, or as file attachments on WhatsApp and CLI. The delivery method depends on the platform and provider:\n\n- Telegram: voice bubbles require Opus format. OpenAI, ElevenLabs, and Mistral produce Opus natively. Other providers require ffmpeg for conversion.\n- Discord: voice bubbles supported natively.\n- WhatsApp: audio sent as file attachments.\n- CLI: audio saved as files.\n\n## Plugin System\n\nThe plugin system coexists with built-in and command providers. Built-in names always win name collisions. Plugins are enabled with hermes plugins enable <name> and configured by setting the provider name in tts.provider or stt.provider.\n\n### Plugin Directory Structure\n\n\n~/.hermes/plugins/<name>/\n├── plugin.yaml\n└── __init__.py\n\n\n### Plugin Registration\n\nFor TTS plugins, implement a register(ctx) function that calls ctx.register_tts_provider(MyTTSProvider()). For STT plugins, implement register(ctx) that calls ctx.register_transcription_provider(MySTTProvider()).\n\n## Environment Variables\n\nThe following environment variables are used for API keys and configuration:\n\n| Variable | Purpose |\n|----------|---------|\n| ELEVENLABS_API_KEY | ElevenLabs TTS/STT |\n| VOICE_TOOLS_OPENAI_KEY | OpenAI TTS/STT |\n| MINIMAX_API_KEY | MiniMax global TTS |\n| MINIMAX_CN_API_KEY | MiniMax China TTS |\n| MISTRAL_API_KEY | Mistral TTS/STT |\n| GEMINI_API_KEY | Google Gemini TTS |\n| XAI_API_KEY | xAI TTS/STT |\n| DEEPINFRA_API_KEY | DeepInfra TTS/STT |\n| NEUTTS_API_KEY | NeuTTS TTS |\n| KITTENTTS_API_KEY | KittenTTS TTS |\n| GROQ_API_KEY | Groq STT |\n| VOLCENGINE_APP_ID | Doubao TTS/STT |\n| VOLCENGINE_ACCESS_TOKEN | Doubao TTS/STT |\n| HERMES_LOCAL_STT_COMMAND | Legacy local STT command |\n| HERMES_LOCAL_STT_LANGUAGE | Legacy local STT language hint |\n\n## Command Provider Security\n\nCommand providers run with user permissions. Treat templates as trusted local input. Hermes secrets are scrubbed from the subprocess environment; PATH, HOME, and locale are kept. Use env_passthrough to allow specific API keys through to the command.\n\n## Process Management\n\nProcess-tree cleanup on timeout applies to both TTS and STT command providers. On Unix, this uses killpg; on Windows, taskkill /T. Shell-quoting is automatic; do not pre-quote placeholder values. Use {{ and }} for literal braces in command templates.\n\n## Nous Portal Integration\n\nNous Portal subscribers can enable gateway tools by running hermes setup --portal to log in. Existing installs use hermes model or hermes tools to select Nous Subscription for TTS.\n\n## Platform-Specific Notes\n\n### Windows\n\nOn Windows, use explicit cmd /c or PowerShell wrapper for shell features in STT commands. The HERMES_LOCAL_STT_COMMAND is tokenized into argv with no shell interpretation.\n\n### macOS\n\nInstall ffmpeg with brew install ffmpeg for Telegram voice bubbles with non-native Opus providers.\n\n### Linux\n\nInstall ffmpeg with sudo apt install ffmpeg (Ubuntu/Debian) or sudo dnf install ffmpeg (Fedora).\n\n## Provider-Specific Notes\n\n### Edge TTS\n\nDefault provider, free, no API key required. Default voice is en-US-AriaNeural.\n\n### ElevenLabs\n\nRequires ELEVENLABS_API_KEY. Voice and model are configurable. Produces Opus natively for Telegram voice bubbles.\n\n### OpenAI\n\nRequires VOICE_TOOLS_OPENAI_KEY. Models include gpt-4o-mini-tts. Produces Opus natively for Telegram voice bubbles.\n\n### MiniMax\n\nRequires MINIMAX_API_KEY (global) or MINIMAX_CN_API_KEY (China). Region selection determines endpoint and credential pair.\n\n### Mistral\n\nRequires MISTRAL_API_KEY. Install with cd ~/.hermes/hermes-agent && uv pip install -e \".[mistral]\". Produces Opus natively for Telegram voice bubbles.\n\n### Google Gemini\n\nRequires GEMINI_API_KEY. Supports persona prompts and audio tags for expressive TTS.\n\n### xAI\n\nRequires XAI_API_KEY. Supports custom voice cloning and audio tags. Voice ID is required.\n\n### DeepInfra\n\nRequires DEEPINFRA_API_KEY.\n\n### NeuTTS\n\nRequires NEUTTS_API_KEY.\n\n### KittenTTS\n\nRequires KITTENTTS_API_KEY.\n\n### Piper\n\nLocal, CPU-only, no API key. First call downloads model to ~/.hermes/cache/piper-voices/. Supports 44 languages.\n\n### Local Whisper\n\nDefault STT provider. Requires faster-whisper. Model sizes: tiny, base (default), small, medium, large-v3.\n\n### Groq\n\nRequires GROQ_API_KEY. Language hint configurable.\n\n### OpenAI STT\n\nRequires VOICE_TOOLS_OPENAI_KEY. Models: whisper-1, gpt-4o-mini-transcribe, gpt-4o-transcribe, gpt-transcribe.\n\n### Mistral STT\n\nRequires MISTRAL_API_KEY. Models: voxtral-mini-latest, voxtral-mini-2602.\n\n### xAI STT\n\nRequires XAI_API_KEY. Model: grok-stt. Language hint defaults to 'en'.\n\n### ElevenLabs STT\n\nRequires ELEVENLABS_API_KEY.\n\n### DeepInfra STT\n\nRequires DEEPINFRA_API_KEY.\n\n### local_command STT\n\nUses HERMES_LOCAL_STT_COMMAND env var with placeholders {input_path}, {output_dir}, {language}, {model}.\n\n## Configuration File Structure\n\nThe configuration file ~/.hermes/config.yaml uses the following structure:\n\nyaml\ntts:\n provider: edge\n speed: 1.0\n edge:\n voice: en-US-AriaNeural\n openai:\n model: gpt-4o-mini-tts\n max_text_length: 8192\n minimax:\n region: global\n gemini:\n audio_tags: true\n persona_prompt_file: ~/.hermes/tts/butler-voice.md\n xai:\n voice_id: nlbqfwie\n auto_speech_tags: true\n language: auto\n speed: 1.0\n optimize_streaming_latency: 0\n sample_rate: 24000\n text_normalization: true\n piper:\n voice: en_US-lessac-medium\n length_scale: 1.0\n noise_scale: 0.667\n noise_w_scale: 0.8\n providers:\n voxcpm:\n command: voxcpm --ref ~/voice.wav --text-file {input_path} --out {output_path}\n output_format: mp3\n timeout: 180\n voice_compatible: true\n max_text_length: 5000\n env_passthrough:\n - VOXCPM_API_KEY\n\nstt:\n provider: local\n language: en\n local:\n model: base\n groq:\n language: en\n openai:\n model: whisper-1\n mistral:\n model: voxtral-mini-latest\n xai:\n model: grok-stt\n language: en\n providers:\n parakeet:\n command: parakeet-asr --model nvidia/parakeet-tdt-0.6b-v2 --in {input_path} --out {output_path}\n format: txt\n language: en\n timeout: 300\n\n\n## Troubleshooting\n\n### TTS Command Provider Fails\n\nCheck that the CLI tool is installed and on PATH. Verify that any required API keys are passed via env_passthrough. Check stderr/stdout for error details.\n\n### STT Command Provider Returns No Output\n\nEnsure the command writes to {output_path} or produces stdout. Verify the format setting matches the output.\n\n### Telegram Voice Bubbles Not Working\n\nEnsure ffmpeg is installed for non-native Opus providers. Check that voice_compatible: true is set for command providers.\n\n### Audio Tags Not Applied\n\nVerify that tts.gemini.audio_tags: true or tts.xai.auto_speech_tags: true is set. Check that the auxiliary model is configured correctly.\n\n### Piper Model Not Downloading\n\nEnsure piper-tts is installed. Check network connectivity. The model downloads to ~/.hermes/cache/piper-voices/.\n\n### MiniMax Region Mismatch\n\nEnsure the region setting matches the credential. Explicit region requires the corresponding API key. base_url override pointing at another region's official endpoint is rejected.\n\n### STT Fallback Not Working\n\nCheck that fallback providers have the required API keys. Verify that the fallback chain is configured correctly. If nothing is available, voice messages pass through with a note.\n\n## Performance Considerations\n\n- Local Whisper model size affects accuracy and speed: tiny is fastest, large-v3 is most accurate.\n- Piper models are downloaded on first use (~20-90MB) and cached locally.\n- Cloud providers may have latency and cost implications.\n- Audio tags add a hidden rewrite pass that may increase response time.\n- Gemini persona prompts add context to TTS requests but may increase token usage.\n\n## Compatibility\n\n- TTS providers deliver audio as voice bubbles on Telegram and Discord, or file attachments on WhatsApp and CLI.\n- STT providers transcribe voice messages into conversation text across all supported platforms.\n- Custom command providers work with any CLI tool that accepts the specified placeholders.\n- Python plugins support full SDK integration with optional hooks for extended functionality." }