Audio and Voice Notes Processing

What it does

When audio understanding is turned on or automatically identified, OpenClaw performs these steps:

It finds the initial audio attachment, whether a local file path or URL, and fetches it if necessary.
Before sending data to any model entry, it enforces maxBytes.
The first qualifying model entry is tried in sequence, either a provider or CLI. If that entry fails or is skipped due to size or timeout constraints, the next entry gets a turn.
Upon a successful run, Body gets replaced by an [Audio] block, and {{Transcript}} is assigned.

Once transcription finishes, CommandBody and RawBody are also populated with the transcript text, ensuring slash commands continue to function. With --verbose enabled, the logs indicate when transcription occurs and when it substitutes the body content.

Auto-detection (default)

If no models have been configured and tools.media.audio.enabled is not set to false, OpenClaw performs auto-detection in the order shown below, stopping at the first option that works:

Active reply model, provided its provider supports audio understanding.
Configured provider auth, any models.providers.* entry whose provider offers audio transcription and has authentication available. This check happens before local CLIs, so a configured API key always takes precedence over a local binary on PATH. When multiple providers are configured, priority is: Groq, OpenAI, xAI, Deepgram, Google, SenseAudio, ElevenLabs, Mistral.
Local CLIs (used only when no provider auth was resolved). OpenClaw assembles an ordered fallback list:
- whisper-cli, which defaults to CPU only if an earlier model invocation in the same process detected Metal or CUDA
- sherpa-onnx-offline running on its default CPU provider, which requires SHERPA_ONNX_MODEL_DIR with tokens.txt, encoder.onnx, decoder.onnx, and joiner.onnx
- whisper-cli when Metal or CUDA is only build-capable or the chosen backend has not been observed
- parakeet-mlx on Apple Silicon (MLX-capable; device usage remains unobserved)
- whisper (a Python CLI that downloads models automatically)

Install or link provenance indicates capability, not execution. It never promotes a candidate ahead of CPU sherpa by itself. During setup or status checks, OpenClaw does not load a model solely to probe a backend. When whisper.cpp is auto-detected, its normal model-run logs stay active so OpenClaw can capture the upstream using … backend line. Explicit CLI entries retain their configured output flags.

The Gemini CLI auto-detect for media understanding has been replaced by a sandboxed Antigravity CLI (agy) fallback for images and video; audio does not use a CLI fallback beyond the local binaries listed above.

To turn off auto-detection, configure tools.media.audio.enabled: false. To customize, add capability-tagged entries to tools.media.models.

Note

Binary detection is best-effort across macOS, Linux, and Windows. Ensure the CLI is on PATH (with ~ expanded), or specify an explicit CLI model with a full command path.

Examine the local selection without performing audio transcription:

openclaw capability audio providers
openclaw doctor --lint --only core/doctor/local-audio-acceleration --severity-min info

The provider inventory reports the local fallback winner separately from the global provider selection, along with capable, requested, and observed backend fields. After transcription runs, /status indicates the requested or observed backend in the media line. Explicit audio-capable tools.media.models CLI entries still bypass auto-selection; use their backend-specific flags such as sherpa --provider=cuda or whisper.cpp --no-gpu and --device.

Config examples

Provider + CLI fallback (OpenAI + Whisper CLI)

{
  tools: {
    media: {
      models: [
        { provider: "openai", model: "gpt-4o-transcribe", capabilities: ["audio"] },
        {
          type: "cli",
          command: "whisper",
          args: ["--model", "base", "{{AttachmentPath}}"],
          timeoutSeconds: 45,
          capabilities: ["audio"],
        },
      ],
      audio: { enabled: true, preferredModel: "openai/gpt-4o-transcribe" },
    },
  },
}

Provider-only (Deepgram)

{
  tools: {
    media: {
      models: [{ provider: "deepgram", model: "nova-3", capabilities: ["audio"] }],
      audio: { enabled: true },
    },
  },
}

Provider-only (Mistral Voxtral)

{
  tools: {
    media: {
      models: [{ provider: "mistral", model: "voxtral-mini-latest", capabilities: ["audio"] }],
      audio: { enabled: true },
    },
  },
}

Provider-only (SenseAudio)

{
  tools: {
    media: {
      models: [
        {
          provider: "senseaudio",
          model: "senseaudio-asr-pro-1.5-260319",
          capabilities: ["audio"],
        },
      ],
      audio: { enabled: true },
    },
  },
}

Echo transcript to chat (opt-in)

{
  tools: {
    media: {
      audio: {
        enabled: true,
        echoTranscript: true,
        echoFormat: '📝 "{transcript}"',
      },
    },
  },
}

Notes and limits

Provider authentication follows the standard model auth sequence: auth profiles, environment variables, and models.providers.*.apiKey.
For Groq configuration, see Groq.
Deepgram reads DEEPGRAM_API_KEY when provider: "deepgram" is in use. Configuration details are at Deepgram.
Mistral setup instructions: Mistral.
SenseAudio picks up SENSEAUDIO_API_KEY when provider: "senseaudio" is active. Setup details: SenseAudio.
Audio providers can rely on defaults under tools.media.audio or override baseUrl, headers, providerOptions, and limits in their tools.media.models[] entry.
The default audio size limit is 20MB. An entry-level maxBytes override can adjust this; oversized audio is skipped for that model and the system tries the next entry.
Audio files smaller than 1024 bytes are discarded before provider or CLI transcription.
The default maxChars for audio is unset, meaning the full transcript is returned. Set tools.media.audio.maxChars or a per-entry maxChars to trim the output.
OpenAI's auto-detect default is gpt-4o-transcribe; use model: "gpt-4o-mini-transcribe" for a more economical or faster option.
Templates receive the transcript as {{Transcript}}.
tools.media.audio.echoTranscript is disabled by default; echoFormat accepts a {transcript} placeholder.
CLI stdout is limited to 5MB; keep CLI output brief.
For CLI args, use {{AttachmentPath}} to point to the local audio file. Run openclaw doctor --fix to migrate deprecated {input} placeholders from older audio.transcription.command configurations (retired key: audio.transcription, replaced by tools.media.models). {{MediaPath}} remains as a deprecated compatibility alias.
tools.media.concurrency limits media tasks; it does not function as a GPU scheduler.

Resident local STT

Auto-detected local STT still operates as a process per request. OpenClaw does not currently host a persistent whisper.cpp server because the standard Homebrew whisper-cpp package disables that server, and the upstream example lacks a configured bounded admission queue. A plugin-owned resident lifecycle requires a maintained packaged worker with health and startup checks, model residency, bounded queuing, cancellation and timeout support, loopback-only no-auth operation, and no cloud fallback before it can be safely enabled.

Proxy environment support

Provider-based audio transcription respects standard outbound proxy environment variables, following undici's EnvHttpProxyAgent semantics:

HTTPS_PROXY / https_proxy
HTTP_PROXY / http_proxy
ALL_PROXY / all_proxy

Lowercase variables take priority over uppercase ones; NO_PROXY/no_proxy entries (hostnames, *.suffix, or host:port) bypass the proxy. When no proxy environment variables are set, direct outbound connections are used. If proxy configuration fails (e.g., a malformed URL), OpenClaw logs a warning and falls back to a direct fetch.

Mention detection in groups

On channels that support audio preflight, OpenClaw transcribes audio before checking for mentions when requireMention: true is set for a group chat. This allows a voice note without a caption to pass the mention gate if its transcript contains a configured mention pattern. Channel-specific documentation describes transports that require a typed mention instead.

How it works:

If a voice message has no text body and the group requires mentions, OpenClaw performs a preflight transcription of the first audio attachment.
The transcript is scanned for mention patterns (for example @BotName, emoji triggers).
If a mention is detected, the message proceeds through the full reply pipeline.

Fallback behavior: if preflight transcription fails (timeout, API error, etc.), the message falls back to text-only mention detection so mixed messages (text + audio) are never dropped.

Opt-out per Telegram group/topic:

Set channels.telegram.groups.<chatId>.disableAudioPreflight: true to skip preflight transcript mention checks for that group.
Set channels.telegram.groups.<chatId>.topics.<threadId>.disableAudioPreflight to override per-topic (true to skip, false to force-enable).
Default is false (preflight enabled when mention-gated conditions match).

Example: a user sends a voice note saying "Hey @Claude, what's the weather?" in a Telegram group with requireMention: true. The voice note is transcribed, the mention is detected, and the agent replies.

Gotchas

Scope rules operate on a first match basis; chatType gets converted into direct, group, or channel.
The CLI must exit with code 0 and output plain text; JSON output requires processing through jq -r .text.
Authoritative file output modes behave as follows: when the inferred transcript file is absent or empty, no transcript is produced, rather than reverting to CLI progress output.
For parakeet-mlx, configure --output-format txt (or all) together with --output-dir and the standard {filename} output template. The upstream PARAKEET_OUTPUT_FORMAT and PARAKEET_OUTPUT_TEMPLATE environment variables are also recognized. OpenClaw reads <output-dir>/<media-basename>.txt; the default srt format, alternative formats, and custom output templates continue to write to stdout.
Set timeouts to reasonable values (timeoutSeconds, 60 seconds by default) so the reply queue isn't blocked.
During preflight transcription, only the first audio attachment is processed for mention detection. Any additional audio attachments are handled later in the main media understanding phase.