Voice Mode in Hermes Agent: CLI, Telegram, Discord Voice Channels

# CLI voice mode (microphone + audio playback)
cd ~/.hermes/hermes-agent && uv pip install -e ".[voice]"

# Discord + Telegram messaging (includes discord.py[voice] for VC support)
cd ~/.hermes/hermes-agent && uv pip install -e ".[messaging]"

# Premium TTS (ElevenLabs)
cd ~/.hermes/hermes-agent && uv pip install -e ".[tts-premium]"

# Local TTS (NeuTTS, optional)
python -m pip install -U neutts[all]

# Everything at once
cd ~/.hermes/hermes-agent && uv pip install -e ".[all]"

# macOS
brew install portaudio ffmpeg opus
brew install espeak-ng # for NeuTTS

# Ubuntu/Debian
sudo apt install portaudio19-dev ffmpeg libopus0
sudo apt install espeak-ng # for NeuTTS

# Speech-to-Text, local provider needs NO key at all
# pip install faster-whisper # Free, runs locally, recommended
GROQ_API_KEY = your-key # Groq Whisper, fast, free tier (cloud)
VOICE_TOOLS_OPENAI_KEY = your-key # OpenAI Whisper, paid (cloud)

# Text-to-Speech (optional, Edge TTS and NeuTTS work without any key)
ELEVENLABS_API_KEY = *** # ElevenLabs, premium quality
# VOICE_TOOLS_OPENAI_KEY above also enables OpenAI TTS

https://discord.com/oauth2/authorize?client_id=YOUR_APP_ID&scope=bot+applications.commands&permissions=309240908864

# macOS (Homebrew)
brew install opus

# Ubuntu/Debian
sudo apt install libopus0

# Discord bot (already configured for text)
DISCORD_BOT_TOKEN = your-bot-token
DISCORD_ALLOWED_USERS = your-user-id

# STT, local provider needs no key (pip install faster-whisper)
# GROQ_API_KEY=your-key # Alternative: cloud-based, fast, free tier

# TTS, optional. Edge TTS and NeuTTS need no key.
# ELEVENLABS_API_KEY=*** # Premium quality
# VOICE_TOOLS_OPENAI_KEY=*** # OpenAI TTS / Whisper

hermes gateway # Start the gateway (connects to configured platforms)
hermes gateway setup # Interactive setup wizard for first-time configuration

Configuration Reference

config.yaml

The following configuration options are available in ~/.hermes/config.yaml:

# Voice recording (CLI)
voice:
  record_key: "ctrl+b" # Key to start/stop recording
  max_recording_seconds: 120 # Maximum recording length
  auto_tts: false # Auto-enable TTS when voice mode starts
  beep_enabled: true # Play record start/stop beeps
  silence_threshold: 200 # RMS level (0-32767) below which counts as silence
  silence_duration: 3.0 # Seconds of silence before auto-stop

# Speech-to-Text
stt:
  enabled: true # set to false to skip auto-transcription, # the gateway still caches the audio file and
                 # passes its path to the agent as part of the
                 # inbound message, useful for custom pipelines
                 # (diarization, alignment, archival, etc.)
  provider: "local" # "local" (free) | "groq" | "openai" | "mistral" | "xai"
  local:
    model: "base" # tiny, base, small, medium, large-v3
    # model: "whisper-1" # Legacy: used when provider is not set

# Text-to-Speech
tts:
  provider: "edge" # "edge" (free) | "elevenlabs" | "openai" | "neutts" | "minimax" | "mistral" | "gemini" | "xai" | "kittentts" | "piper"
  edge:
    voice: "en-US-AriaNeural" # 322 voices, 74 languages
  elevenlabs:
    voice_id: "pNInz6obpgDQGcFmaJgB" # Adam
    model_id: "eleven_multilingual_v2"
  openai:
    model: "gpt-4o-mini-tts"
    voice: "alloy" # alloy, echo, fable, onyx, nova, shimmer
    base_url: "https://api.openai.com/v1" # optional: override for self-hosted or OpenAI-compatible endpoints
  neutts:
    ref_audio: ''
    ref_text: ''
    model: neuphonic/neutts-air-q4-gguf
    device: cpu

Environment Variables

Variable	Purpose	Required
GROQ_API_KEY	Groq Whisper STT (fast, free tier)	No (if using local STT)
VOICE_TOOLS_OPENAI_KEY	OpenAI Whisper STT and OpenAI TTS	No (if using local STT/TTS)
STT_GROQ_MODEL	Override default Groq STT model (e.g., whisper-large-v3-turbo)	No
STT_OPENAI_MODEL	Override default OpenAI STT model (e.g., whisper-1)	No
GROQ_BASE_URL	Custom Groq endpoint	No
STT_OPENAI_BASE_URL	Custom OpenAI STT endpoint	No
ELEVENLABS_API_KEY	ElevenLabs TTS (premium quality)	No (if using Edge TTS or NeuTTS)
DISCORD_BOT_TOKEN	Discord bot token	Yes (for Discord)
DISCORD_ALLOWED_USERS	Comma-separated user IDs or usernames allowed to interact via voice	Yes (for Discord voice channels)
DISCORD_REQUIRE_MENTION	Set to false to disable @mention requirement in server channels	No
DISCORD_FREE_RESPONSE_CHANNELS	Comma-separated channel IDs where no @mention is needed	No

STT Provider Comparison

Provider	Model	Speed	Quality	Cost	API Key Required
Local	base	Fast (depends on CPU/GPU)	Good	Free	No
Local	small	Medium	Better	Free	No
Local	large-v3	Slow	Best	Free	No
Groq	whisper-large-v3-turbo	Very fast (~0.5s)	Good	Free tier	Yes
Groq	whisper-large-v3	Fast (~1s)	Better	Free tier	Yes
OpenAI	whisper-1	Fast (~1s)	Good	Paid	Yes
OpenAI	gpt-4o-transcribe	Medium (~2s)	Best	Paid	Yes
Mistral	voxtral-mini-latest	Fast	Good	Paid	Yes
xAI	grok-stt	Fast	Good	Paid	Yes

Provider priority (automatic fallback): local > groq > openai.

TTS Provider Comparison

Provider	Quality	Cost	Latency	Key Required
Edge TTS	Good	Free	~1s	No
ElevenLabs	Excellent	Paid	~2s	Yes
OpenAI TTS	Good	Paid	~1.5s	Yes
NeuTTS	Good	Free	Depends on CPU/GPU	No

NeuTTS uses the tts.neutts config block in config.yaml.

How It Works

CLI Voice Mode

Start the CLI with hermes and enable voice mode with /voice on. Press Ctrl+B (configurable via voice.record_key in config.yaml, default ctrl+b). A beep plays (880Hz) and recording starts. A live audio level bar shows your input: ● [▁▂▃▅▇▇▅▂] ❯. After 3 seconds of silence (configurable via voice.silence_duration), recording auto-stops. Two beeps play (660Hz) confirming the recording ended. The audio is transcribed via Whisper and sent to the agent. If TTS is enabled, the agent's reply is spoken aloud. Recording automatically restarts, allowing you to speak again without pressing any key. This loop continues until you press Ctrl+B during recording (exits continuous mode) or 3 consecutive recordings detect no speech.

Silence Detection

The two-stage algorithm works as follows:

• Speech confirmation: Waits for audio above the RMS threshold (200, configurable via voice.silence_threshold) for at least 0.3 seconds, tolerating brief dips between syllables.
• End detection: Once speech is confirmed, triggers after 3.0 seconds of continuous silence (configurable via voice.silence_duration).

If no speech is detected at all for 15 seconds, recording stops automatically. Both silence_threshold and silence_duration are configurable in config.yaml. You can also disable the record start/stop beeps with voice.beep_enabled: false.

Streaming TTS

When TTS is enabled, the agent speaks its reply sentence-by-sentence as it generates text. This works with every TTS provider. The pipeline buffers text deltas into complete sentences (minimum 20 characters), strips markdown formatting, emoji, and code blocks, then plays audio per sentence in real-time. Providers with a chunked PCM API (ElevenLabs, OpenAI) stream raw audio for the lowest time-to-first-word. Every other provider (including the default Edge) synthesizes and plays each sentence as it completes. The same pipeline runs in the classic CLI, the TUI, and the desktop app. In a desktop voice conversation, the reply text is fed live into a per-reply speech WebSocket as the model generates it, so speech overlaps generation with one socket and one audio clock per reply, no per-sentence connection gaps.

Barge-in

You can interrupt the agent mid-speech in two ways:

• Talk over it: In continuous voice mode, a voice-activity monitor listens while the agent speaks and cuts playback the moment you start talking, then goes straight back to recording. The detector calibrates its noise floor against the playback itself, so speaker bleed does not self-trigger. Disable with voice.barge_in: false in config.yaml.
• Type or press the record key: Sending a new message or hitting the push-to-talk key stops playback instantly on every surface.

The agent knows it was interrupted: the next message carries a short note telling the model its spoken reply was cut off, so it can react naturally (e.g., "rude!") or pick up where it left off instead of being oblivious.

Hallucination Filter

Whisper sometimes generates phantom text from silence or background noise (e.g., "Thank you for watching", "Subscribe"). The agent filters these out using a set of 26 known hallucination phrases across multiple languages, plus a regex pattern that catches repetitive variations.

Discord Voice Channels

When the bot joins a voice channel, it listens to each user's audio stream independently. It detects silence: 1.5 seconds of silence after at least 0.5 seconds of speech triggers processing. The audio is transcribed via Whisper STT (local, Groq, or OpenAI), then processed through the full agent pipeline (session, tools, memory). The reply is spoken back in the voice channel via TTS.

Transcripts appear in the text channel where /voice join was issued, formatted as [Voice] @user: what you said. Agent responses are sent as text in the channel AND spoken in the VC. The bot automatically pauses its audio listener while playing TTS replies, preventing it from hearing and re-processing its own output (echo prevention).

Discord: Channels vs DMs

The bot supports two interaction modes on Discord:

• Direct Message (DM): Open the bot's profile and click "Message". No @mention needed. Voice replies and all commands work the same as in channels. Recommended for personal use.
• Server Channel: Type in a text channel where the bot is present. You must @mention the bot (e.g., @hermesbyt4 hello). Make sure you select the bot user from the mention popup, not the role with the same name.

To disable the mention requirement in server channels, add to ~/.hermes/.env:

DISCORD_REQUIRE_MENTION = false

Or set specific channels as free-response (no mention needed):

DISCORD_FREE_RESPONSE_CHANNELS = 123456789,987654321

Gateway Voice Reply (Telegram and Discord)

The following commands work in both Telegram and Discord (DMs and text channels):

• /voice: Toggle voice mode on/off.
• /voice on: Voice replies only when you send a voice message.
• /voice tts: Voice replies for ALL messages.
• /voice off: Disable voice replies.
• /voice status: Show current setting.

The voice mode setting is persisted across gateway restarts.

Modes

Mode	Command	Behavior
off	/voice off	Text only (default)
voice_only	/voice on	Speaks reply only when you send a voice message
all	/voice tts	Speaks reply to every message

Platform Delivery

• Telegram: Voice bubble (Opus/OGG). Plays inline in chat. ffmpeg converts MP3 to Opus if needed.
• Discord: Native voice bubble (Opus/OGG). Plays inline like a user voice message. Falls back to file attachment if voice bubble API fails.

Discord Voice Channel Commands

Use these in the Discord text channel where the bot is present:

• /voice join: Bot joins your current voice channel.
• /voice channel: Alias for /voice join.
• /voice leave: Bot disconnects from voice channel.
• /voice status: Show voice mode and connected channel.

You must be in a voice channel before running /voice join. The bot joins the same VC you are in.

Access Control

Only users listed in DISCORD_ALLOWED_USERS can interact via voice. Other users' audio is silently ignored.

# ~/.hermes/.env
DISCORD_ALLOWED_USERS = 284102345871466496

Requirements and Limitations

Prerequisites

• Hermes Agent installed via the install script.
• An LLM provider configured (via hermes model or ~/.hermes/.env).
• A working base setup (agent responds to text).

Supported Platforms

• CLI: Classic CLI (hermes chat) and TUI (hermes --tui).
• Telegram: Via the gateway.
• Discord: Text channels, DMs, and voice channels.

Python Version

Hermes Agent uses uv for package management. The specific Python version is not stated in the sources, but the installation commands assume a working Python environment with uv.

Known Limitations

• Discord voice channels: Only users listed in DISCORD_ALLOWED_USERS can interact via voice. Other users' audio is silently ignored.
• Server Members Intent: Only needed if DISCORD_ALLOWED_USERS uses usernames. If numeric user IDs are used, it can be left off.
• Opus codec: Must be installed on the machine running the gateway. Auto-loads from specific paths on macOS and Linux.
• Docker: If running Hermes inside Docker on a Linux desktop, the container needs access to the host audio socket. See the Docker audio bridge notes for a PulseAudio/PipeWire-compatible setup.
• Hallucination Filter: Catches most cases automatically, but if phantom transcripts persist, a quieter environment, adjusting silence_threshold, or using a different STT model may help.
• Barge-in: Can be disabled with voice.barge_in: false in config.yaml.
• Recording: Maximum recording length is configurable via voice.max_recording_seconds (default 120).
• STT Provider Priority: Automatic fallback order is local > groq > openai.
• TTS: Edge TTS is the default fallback if no API key is configured.

Troubleshooting

"No audio device found" (CLI)

PortAudio is not installed. Install it:

brew install portaudio # macOS
sudo apt install portaudio19-dev # Ubuntu

If running Hermes inside Docker on a Linux desktop, the container also needs access to your host audio socket. See the Docker audio bridge notes for a PulseAudio/PipeWire-compatible setup.

Bot doesn't respond in Discord server channels

The bot requires an @mention by default in server channels. Make sure you:

• Type @ and select the bot user (with the discriminator), not the role with the same name.
• Or use DMs instead (no mention needed).
• Or set DISCORD_REQUIRE_MENTION=false in ~/.hermes/.env.

Bot joins VC but doesn't hear me

• Check your Discord user ID is in DISCORD_ALLOWED_USERS.
• Make sure you are not muted in Discord.
• The bot needs a SPEAKING event from Discord before it can map your audio. Start speaking within a few seconds of joining.

Bot hears me but doesn't respond

• Verify STT is available: install faster-whisper (no key needed) or set GROQ_API_KEY / VOICE_TOOLS_OPENAI_KEY.
• Check the LLM model is configured and accessible.
• Review gateway logs: tail -f ~/.hermes/logs/gateway.log.

Bot responds in text but not in voice channel

• TTS provider may be failing. Check API key and quota.
• Edge TTS (free, no key) is the default fallback.
• Check logs for TTS errors.

Whisper returns garbage text

The hallucination filter catches most cases automatically. If you are still getting phantom transcripts:

• Use a quieter environment.
• Adjust silence_threshold in config (higher = less sensitive).
• Try a different STT model.