Media Understanding for Inbound Image, Audio, and Video

OpenClaw can process incoming media (image/audio/video) by generating summaries before the reply pipeline starts, which lets command parsing and routing work from concise text rather than raw data. The understanding layer automatically detects local tools or provider credentials, though you can also specify models explicitly. Original media still reaches the model as normal; if understanding is turned off or fails, the reply flow proceeds without any changes.

Capability metadata is registered by vendor plugins, covering which provider handles which media type, the default model, and priority. The shared tools.media configuration, fallback ordering, and reply-pipeline integration are managed by OpenClaw core.

How it works

Collect attachments

Gather ordered inbound media details (path, url, contentType, and kind).

Select per capability

For every enabled capability (image/audio/video), pick attachments according to the attachments policy (default: only the first attachment).

Choose a model

Choose the first suitable model entry (based on size, capability, and available auth).

Fall back on failure

When a model errors, times out, or the media goes beyond maxBytes, move to the next entry.

Apply on success

Body turns into an [Image], [Audio], or [Video] block. Audio additionally sets {{Transcript}}; command parsing uses the caption text when it exists, otherwise the transcript. Captions stay as User text: within the block.

Config

tools.media contains one capability-tagged model list plus small per-capability controls:

{
  tools: {
    media: {
      concurrency: 2, // max concurrent capability runs (default)
      models: [
        { provider: "openai", model: "gpt-4o-mini-transcribe", capabilities: ["audio"] },
        { provider: "google", model: "gemini-3-flash-preview", capabilities: ["image", "video"] },
      ],
      image: { preferredModel: "google/gemini-3-flash-preview" },
      audio: { enabled: true },
      video: { enabled: true },
    },
  },
}

Per-capability (image/audio/video) keys:

Prompts, limits, language hints, request overrides, and provider options can be set as capability defaults or overridden on individual tools.media.models[] entries. Capability defaults also apply to auto-detected providers when no explicit model is configured.

Model entries

Each models[] entry is either a provider entry (default) or a CLI entry:

Provider entry

{
  type: "provider", // default if omitted
  provider: "openai",
  model: "gpt-5.6-sol",
  prompt: "Describe the image in <= 500 chars.",
  maxChars: 500,
  maxBytes: 10485760,
  timeoutSeconds: 60,
  capabilities: ["image"],
  profile: "vision-profile",
  preferredProfile: "vision-fallback",
}

CLI entry

{
  type: "cli",
  command: "gemini",
  args: [
    "-m",
    "gemini-3-flash",
    "--allowed-tools",
    "read_file",
    "Read the media at {{AttachmentPath}} and describe it in <= {{MaxChars}} characters.",
  ],
  maxChars: 500,
  maxBytes: 52428800,
  timeoutSeconds: 120,
  capabilities: ["video", "image"],
}

Within CLI templates, {{AttachmentUrl}}, {{AttachmentContentType}}, {{AttachmentDir}}, {{AttachmentIndex}}, {{OutputDir}} (the scratch directory created for this run) and {{OutputBase}} (scratch file base path without extension) are also available. The legacy names {{MediaPath}}, {{MediaUrl}}, {{MediaType}}, and {{MediaDir}} persist only as deprecated aliases for compatibility.

Provider credentials

Media understanding for providers resolves authentication exactly like standard model calls: auth profiles, then environment variables, and finally models.providers.<providerId>.apiKey. Entries under tools.media.models[] cannot carry an inline apiKey field.

{
  models: {
    providers: {
      openai: { apiKey: "<OPENAI_API_KEY>" },
      moonshot: { apiKey: "<MOONSHOT_API_KEY>" },
    },
  },
}

For details on profiles, environment variables, and custom base URLs, refer to Tools and custom providers.

Rules and behavior

• When media goes past maxBytes, that model is skipped and the next one gets a chance.
• Audio shorter than 1024 bytes counts as empty or corrupt and is skipped before transcription; the agent receives a fixed placeholder transcript instead.
• If the currently active primary image model already understands vision natively, OpenClaw omits the [Image] summary block and feeds the raw image straight to the model. MiniMax breaks this rule: minimax, minimax-cn, minimax-portal, and minimax-portal-cn always send image understanding through the plugin-owned MiniMax-VL-01 media provider, even when legacy MiniMax M2.x chat metadata claims image support (only MiniMax-M3 and newer count as natively vision-capable).
• For a text-only Gateway/WebChat primary model, image attachments stay intact as offloaded media://inbound/* refs, so image/PDF tools or a configured image model can still examine them rather than dropping the attachment.
• An explicit openclaw infer image describe --file <path> --model <provider/model> (alias: openclaw capability image describe) invokes that image-capable provider/model directly, including Ollama refs like ollama/qwen2.5vl:7b when a matching image-capable model exists under models.providers.ollama.models[].
• If <capability>.enabled is anything other than false but no models are set up, OpenClaw falls back to the active reply model when its provider handles the capability.

Auto-detect (default)

When tools.media.<capability>.enabled is not false and no models are configured, OpenClaw works through these options in order, stopping at the first one that succeeds:

Configured image model (image only)

agents.defaults.imageModel primary/fallback refs, except when the active reply model already has native vision support. provider/model refs are preferred; bare refs get qualified from configured image-capable provider model entries only if the match is unambiguous.

Active reply model

The active reply model, assuming its provider supports the capability.

Provider auth (audio only, before local CLIs)

Configured models.providers.* entries that handle audio take precedence over local CLIs. Bundled providers are tried in this priority order (alphabetical by provider id breaks ties): Groq/OpenAI → xAI → Deepgram → OpenRouter → Google/SenseAudio → Deepinfra/ElevenLabs → Mistral.

Local CLIs (audio only)

Ready local binaries form an ordered fallback list:

• whisper-cli ranks first, but only after an earlier model invocation in this process detected Metal or CUDA
• CPU-default sherpa-onnx-offline (needs SHERPA_ONNX_MODEL_DIR with tokens.txt/encoder.onnx/decoder.onnx/joiner.onnx)
• whisper-cli when acceleration is merely build-capable or never observed
• parakeet-mlx on Apple Silicon (MLX-capable, device use not observed)
• whisper (Python CLI; uses the turbo model by default, downloads automatically)

Backend capability checks are cached and never load a model. Build capability, requested backend flags, and the backend observed from an actual invocation stay separate. Auto-detected whisper.cpp leaves model-run logs on so the upstream selected-backend line can be captured. Explicit CLI entries keep their configured order, backend flags, and output flags.

Provider auth (image/video)

Configured models.providers.* entries that support the capability are tried before the bundled fallback order. Image-only config providers with an image-capable model register themselves for media understanding automatically, even when they are not a bundled vendor plugin.

Bundled providers are tried in this priority order (alphabetical by provider id breaks ties):

• Image: Anthropic/OpenAI → Google → MiniMax → Deepinfra → MiniMax Portal → Z.AI
• Video: Google → Qwen → Moonshot

To turn off auto-detection for a capability:

{
  tools: {
    media: {
      audio: {
        enabled: false,
      },
    },
  },
}

Note

Binary detection is best-effort across macOS/Linux/Windows; make sure the CLI is on PATH (~ gets expanded), or define an explicit CLI model entry with a full command path.

Proxy support (audio/video provider calls)

Provider-based audio and video understanding respects the standard outbound proxy environment variables, including the NO_PROXY/no_proxy bypass rules: HTTPS_PROXY, HTTP_PROXY, ALL_PROXY, https_proxy, http_proxy, all_proxy. Uppercase variables take a back seat to their lowercase counterparts. When no variables are configured, media understanding connects directly; if the proxy value is invalid, OpenClaw emits a warning and reverts to a direct fetch. Image understanding bypasses this proxy path entirely.

Capabilities

To limit a models[] entry to particular media types, set capabilities on it. For shared lists, OpenClaw applies defaults per bundled provider:

For CLI entries, set capabilities explicitly to prevent unexpected matches; without it, the entry qualifies for every capability list where it appears.

Provider support matrix

Note

MiniMax note: minimax, minimax-cn, minimax-portal, and minimax-portal-cn image understanding always comes from the plugin-owned MiniMax-VL-01 media provider, even if legacy MiniMax M2.x chat metadata claims image input.

Model selection guidance

• When quality and safety are priorities, choose the strongest current-generation model for each media capability.
• For tool-enabled agents handling untrusted inputs, steer clear of older or weaker media models.
• Maintain at least one fallback per capability to ensure availability (a quality model plus a faster or cheaper one).
• CLI fallbacks (whisper-cli, whisper, gemini) step in when provider APIs are down.
• Known file-output modes take precedence: an empty or missing inferred transcript file yields no transcript, not a fallback to CLI progress output.
• parakeet-mlx: pair --output-format txt (or all) with --output-dir and the default {filename} output template. The upstream PARAKEET_OUTPUT_FORMAT and PARAKEET_OUTPUT_TEMPLATE environment variables are also respected. OpenClaw reads <output-dir>/<media-basename>.txt; the default srt format, other formats, and custom output templates still write to stdout.

Attachment policy

Per-capability attachments decides which attachments get processed:

• mode (first" | "all, default: first), Process only the first selected attachment, or all of them.

• maxAttachments (number, default: 1), Cap the number processed.

• prefer (first" | "last" | "path" | "url), Selection preference among candidate attachments.

When mode: "all", outputs are labeled [Image 1/2], [Audio 2/2], etc.

File-attachment extraction

• Every file attached to an incoming message ends up as a model-visible block. Attachments that go through image, audio, or video understanding fall outside this rule, since those stages manage their own results.
• Extracted text from a file is treated as untrusted external content and added to the media prompt, wrapped with boundary markers like <<<EXTERNAL_UNTRUSTED_CONTENT id="...">>> / <<<END_EXTERNAL_UNTRUSTED_CONTENT id="...">>> and a Source: External metadata line.
• This route deliberately skips the long SECURITY NOTICE: banner to keep the media prompt concise, though the boundary markers and metadata still apply.
• Unsupported files saved locally get self-serve guidance only when the reply runtime proves it can read host-local paths (currently non-sandboxed embedded sessions). That path is fenced as untrusted external metadata; the trusted guidance tells the agent to extract the file with its own tools, and modern Office files receive an unzip hint. Generic ACP backends, URL-only attachments, and sandboxed sessions keep the plain [Unsupported document format: <mime>. PDF and plain-text attachments can be read.] marker.
• Files rejected by an operator-configured allowlist never include the self-serve path; a policy rejection must not coach the agent around the operator's decision.
• Files rejected by an operator-configured allowedMimes list get [Attachment type not allowed: <mime>] instead, so the prompt never claims support the active configuration disables.
• Read failures get [Attachment could not be read].
• URL attachments get [Attachment skipped: URL file sources are disabled] when URL file sources are disabled.
• A file with no extractable text gets [No extractable text].
• At most five skip markers render per message; further skipped attachments collapse into one reason-neutral [<n> more attachments skipped] summary so junk attachments cannot grow the prompt without bound. File and image, audio, or video markers share this five-marker budget.
• If a PDF falls back to rendered page images, OpenClaw forwards those images to vision-capable reply models and keeps the placeholder [PDF content rendered to images] in the file block.
• Image, audio, and video decisions record one closed disposition for every attachment candidate: handled, handed to native vision, not selected after the attachment limit, disabled, missing a model, denied by chat scope, or failed.
• Unhandled media gets a bounded model-visible marker. Images handed to native vision and media turns owned by another harness do not add markers.

Config examples

Shared models + overrides

{
  tools: {
    media: {
      models: [
        { provider: "openai", model: "gpt-5.6-sol", capabilities: ["image"] },
        {
          provider: "google",
          model: "gemini-3-flash-preview",
          capabilities: ["image", "audio", "video"],
        },
        {
          type: "cli",
          command: "gemini",
          args: [
            "-m",
            "gemini-3-flash",
            "--allowed-tools",
            "read_file",
            "Read the media at {{AttachmentPath}} and describe it in <= {{MaxChars}} characters.",
          ],
          capabilities: ["image", "video"],
        },
      ],
      audio: {
        attachments: { mode: "all", maxAttachments: 2 },
      },
      video: {
        maxChars: 500,
      },
    },
  },
}

Audio + video only

{
  tools: {
    media: {
      models: [
        {
          provider: "openai",
          model: "gpt-4o-mini-transcribe",
          capabilities: ["audio"],
        },
        {
          type: "cli",
          command: "whisper",
          args: ["--model", "base", "{{AttachmentPath}}"],
          capabilities: ["audio"],
        },
        {
          provider: "google",
          model: "gemini-3-flash-preview",
          capabilities: ["video"],
        },
        {
          type: "cli",
          command: "gemini",
          args: [
            "-m",
            "gemini-3-flash",
            "--allowed-tools",
            "read_file",
            "Read the media at {{AttachmentPath}} and describe it in <= {{MaxChars}} characters.",
          ],
          capabilities: ["video"],
        },
      ],
      audio: {
        enabled: true,
      },
      video: {
        enabled: true,
        maxChars: 500,
      },
    },
  },
}

Image only

{
  tools: {
    media: {
      models: [
        { provider: "openai", model: "gpt-5.6-sol", capabilities: ["image"] },
        { provider: "anthropic", model: "claude-opus-5", capabilities: ["image"] },
        {
          type: "cli",
          command: "gemini",
          args: [
            "-m",
            "gemini-3-flash",
            "--allowed-tools",
            "read_file",
            "Read the media at {{AttachmentPath}} and describe it in <= {{MaxChars}} characters.",
          ],
          capabilities: ["image"],
        },
      ],
      image: {
        enabled: true,
        maxBytes: 10485760,
        maxChars: 500,
      },
    },
  },
}

Multi-modal single entry

{
  tools: {
    media: {
      models: [
        {
          provider: "google",
          model: "gemini-3.1-pro-preview",
          capabilities: ["image", "video", "audio"],
        },
      ],
    },
  },
}

Status output

When media understanding runs, /status includes a per-capability summary line:

📎 Media: image ok (openai/gpt-5.6-sol) · audio ok (whisper-cli observed=metal)

For preflight inventory, run openclaw capability audio providers. Local rows show the local fallback winner separately from global provider selection, readiness, and separate capable/requested/observed backend fields. The same local selection is available as an informational doctor finding:

openclaw doctor --lint --only core/doctor/local-audio-acceleration --severity-min info

Notes

• Understanding is best-effort. Errors do not block replies.
• Attachments are still passed to models even when understanding is disabled.
• Use scope to limit where understanding runs (for example, only DMs).

Related

• Configuration
• Image & media support