Audio & Voice Notes Handling

This page covers inbound transcription and voice-note handling. For inline audio and video players in OpenClaw chat clients, see Media playback.

What it does

When audio understanding is turned on, or picked up automatically, OpenClaw performs these steps:

It finds the first audio attachment, whether a local path or URL, and fetches it if necessary.
Before each model entry receives the data, maxBytes is enforced.
The first eligible model entry, either provider or CLI, is tried in sequence; if that entry fails or declines due to size or timeout, the next one gets a chance.
Once successful, Body gets swapped out for an [Audio] block, and {{Transcript}} is assigned.

After a successful transcription, CommandBody and RawBody are populated with the transcript as well, so slash commands keep functioning. When --verbose is enabled, the logs indicate both when transcription executes and when it takes over the body.

Auto-detection (default)

If no models are configured and tools.media.audio.enabled is not set to false, OpenClaw performs auto-detection in this order, stopping at the first option that works:

Active reply model, provided its provider supports audio understanding.
Configured provider auth: any models.providers.* entry that has auth available for a provider capable of audio transcription. This check happens before local CLIs, meaning a configured API key always takes precedence over a local binary on PATH. When multiple providers are set up, the priority order is: Groq, OpenAI, xAI, Deepgram, Google, SenseAudio, ElevenLabs, Mistral.
Local CLIs (used only when no provider auth is resolved). OpenClaw assembles an ordered fallback list:
- whisper-cli, which precedes CPU defaults only when an earlier model invocation in the current process observed Metal or CUDA
- sherpa-onnx-offline on its default CPU provider, which requires SHERPA_ONNX_MODEL_DIR along with tokens.txt, encoder.onnx, decoder.onnx, and joiner.onnx
- whisper-cli when Metal or CUDA is only build-capable, or when the selected backend is otherwise unobserved
- parakeet-mlx on Apple Silicon, which is MLX-capable, though device usage remains unobserved
- whisper, a Python CLI that downloads models automatically

Install or link provenance counts as capability evidence, not execution proof. On its own, it never advances a candidate ahead of CPU sherpa. OpenClaw does not load a model during setup or status checks merely to probe a backend. Auto-detected whisper.cpp keeps its standard model-run logs active so OpenClaw can capture the upstream using … backend line. Explicit CLI entries retain their configured output flags.

Gemini CLI and Antigravity are excluded from auto-detection for media understanding. Audio does not rely on a CLI fallback beyond the local binaries listed above.

To turn off auto-detection, set tools.media.audio.enabled: false. For customization, add capability-tagged entries to tools.media.models.

Note

Binary detection is best-effort on macOS, Linux, and Windows. Confirm the CLI is on PATH (with ~ expanded), or specify an explicit CLI model using a full command path.

Review the local selection without transcribing any audio:

openclaw capability audio providers
openclaw doctor --lint --only core/doctor/local-audio-acceleration --severity-min info

The provider inventory reports the local fallback winner separately from global provider selection, and also includes capable, requested, and observed backend fields. After transcription completes, /status shows the requested or observed backend in the media line. Explicit audio-capable tools.media.models CLI entries still bypass auto-selection; use their backend-specific flags, such as sherpa --provider=cuda or whisper.cpp --no-gpu and --device.

Config examples

Provider + CLI fallback (OpenAI + Whisper CLI)

{
  tools: {
    media: {
      models: [
        { provider: "openai", model: "gpt-4o-transcribe", capabilities: ["audio"] },
        {
          type: "cli",
          command: "whisper",
          args: ["--model", "base", "{{AttachmentPath}}"],
          timeoutSeconds: 45,
          capabilities: ["audio"],
        },
      ],
      audio: { enabled: true, preferredModel: "openai/gpt-4o-transcribe" },
    },
  },
}

Provider-only (Deepgram)

{
  tools: {
    media: {
      models: [{ provider: "deepgram", model: "nova-3", capabilities: ["audio"] }],
      audio: { enabled: true },
    },
  },
}

Provider-only (Mistral Voxtral)

{
  tools: {
    media: {
      models: [{ provider: "mistral", model: "voxtral-mini-latest", capabilities: ["audio"] }],
      audio: { enabled: true },
    },
  },
}

Provider-only (SenseAudio)

{
  tools: {
    media: {
      models: [
        {
          provider: "senseaudio",
          model: "senseaudio-asr-pro-1.5-260319",
          capabilities: ["audio"],
        },
      ],
      audio: { enabled: true },
    },
  },
}

Echo transcript to chat (opt-in)

{
  tools: {
    media: {
      audio: {
        enabled: true,
        echoTranscript: true,
        echoFormat: '📝 "{transcript}"',
      },
    },
  },
}

Notes and limits

Provider authentication follows the standard model auth order (auth profiles, environment variables, models.providers.*.apiKey).

Groq configuration is documented at Groq.

When provider: "deepgram" is in use, Deepgram will automatically read DEEPGRAM_API_KEY. See Deepgram for setup instructions.

Mistral setup is covered at Mistral.

SenseAudio reads SENSEAUDIO_API_KEY whenever provider: "senseaudio" is active. Configuration details are at SenseAudio.

Audio providers may rely on defaults found under tools.media.audio, or they can override baseUrl, headers, providerOptions, and set limits within their tools.media.models[] entry.

The default audio size cap is 20MB. An entry-level maxBytes override can adjust this; any audio exceeding the limit is skipped for that model, and the next entry gets tried instead.

Audio files smaller than 1024 bytes are filtered out before provider or CLI transcription begins.

By default, maxChars for audio is unset, meaning the full transcript is returned. To trim the output, configure tools.media.audio.maxChars or the per-entry maxChars setting.

OpenAI's auto-detect default is gpt-4o-transcribe; choose model: "gpt-4o-mini-transcribe" for a more economical and faster alternative.

Templates can access the transcript through {{Transcript}}.

tools.media.audio.echoTranscript starts disabled; echoFormat takes a {transcript} placeholder.

CLI stdout is limited to 5MB, so keep command output brief.

For CLI args, point to the local audio file path with {{AttachmentPath}}. Run openclaw doctor --fix to migrate deprecated {input} placeholders from older audio.transcription.command configurations (the retired key is audio.transcription, now replaced by tools.media.models). {{MediaPath}} still works as a deprecated compatibility alias.

tools.media.concurrency constrains media tasks; it does not act as a GPU scheduler.

Resident local STT

Auto-detected local STT still runs as a process per request. OpenClaw does not currently host a persistent whisper.cpp server, since the standard Homebrew whisper-cpp package disables that server, and the upstream example lacks a configured bounded admission queue. Before a plugin-owned resident lifecycle can be safely enabled, it needs a maintained packaged worker with health/startup checks, model residency, bounded queueing, cancellation/timeout, loopback-only no-auth operation, and no cloud fallback.

Proxy environment support

Provider-based audio transcription respects standard outbound proxy environment variables, matching undici's EnvHttpProxyAgent semantics:

HTTPS_PROXY / https_proxy
HTTP_PROXY / http_proxy
ALL_PROXY / all_proxy

Lowercase variables take precedence over uppercase; NO_PROXY/no_proxy entries (hostnames, *.suffix, or host:port) bypass the proxy. When no proxy env vars are present, direct egress is used. If proxy setup fails (e.g., a malformed URL), OpenClaw logs a warning and falls back to direct fetch.

Mention detection in groups

On channels that support audio preflight, when requireMention: true is set for a group chat, OpenClaw transcribes audio before checking for mentions. This allows a captionless voice note to pass the mention gate if its transcript contains a configured mention pattern. Channel-specific docs describe transports that require a typed mention instead.

How it works:

If a voice message lacks a text body and the group requires mentions, OpenClaw runs a preflight transcription on the first audio attachment.
The transcript is scanned for mention patterns (such as @BotName or emoji triggers).
When a mention is found, the message moves through the full reply pipeline.

Fallback behavior: if preflight transcription fails (timeout, API error, etc.), the message reverts to text-only mention detection, so mixed messages (text + audio) are never dropped.

Opt-out per Telegram group/topic:

Configure channels.telegram.groups.<chatId>.disableAudioPreflight: true to bypass preflight transcript mention checks for that group.
Use channels.telegram.groups.<chatId>.topics.<threadId>.disableAudioPreflight to override per-topic (true to skip, false to force-enable).
The default is false (preflight enabled when mention-gated conditions match).

Example: in a Telegram group with requireMention: true, a user sends a voice note saying "Hey @Claude, what's the weather?". The voice note gets transcribed, the mention is detected, and the agent replies.

Gotchas

Scope rules apply on a first-match-wins basis; chatType gets converted into direct, group, or channel.
Make sure your CLI exits with code 0 and outputs plain text; JSON results need to be processed through jq -r .text.
File-output modes that are known take precedence: if the inferred transcript file is missing or empty, no transcript is produced, rather than reverting to CLI progress output.
For parakeet-mlx, go with --output-format txt (or all) together with --output-dir and the standard {filename} output template. The upstream PARAKEET_OUTPUT_FORMAT and PARAKEET_OUTPUT_TEMPLATE environment variables are respected as well. OpenClaw checks <output-dir>/<media-basename>.txt; the default srt format, other formats, and custom output templates continue using stdout.
Set timeouts to sensible values (timeoutSeconds, 60s by default) so the reply queue doesn't get blocked.
During preflight transcription, only the first audio attachment is scanned for mention detection. Any additional audio attachments get handled in the main media-understanding phase.