Media Capabilities Overview: Image, Video, Music, Speech

OpenClaw produces images, video, and music, interprets media it receives (images, audio, video), and reads replies out loud through text-to-speech. Every media capability runs as a tool: the agent chooses when to invoke one based on the conversation, and a tool shows up only once at least one provider behind it is configured.

Live speech follows the Talk session contract rather than the one-shot media tool path. Three modes exist for Talk: provider-native realtime, local or streaming stt-tts, and transcription for speech capture in observe-only mode. Provider catalogs, event envelopes, and cancellation semantics are shared by those modes with telephony, meetings, browser realtime, and native push-to-talk clients.

Capabilities

• Image generation, Produce and edit images from text prompts or reference images with image_generate. In chat sessions this is async: it runs in the background and posts the result once ready.

• Video generation, Text-to-video, image-to-video, and video-to-video through video_generate. Async: it runs in the background and posts the result once ready.

• Music generation, Create music or audio tracks with music_generate. Async in chat sessions, on the shared media-generation task lifecycle.

• Text-to-speech, Turn outbound replies into spoken audio using the tts tool together with tts config. Synchronous.

• Media understanding, Summarize inbound images, audio, and video with vision-capable model providers and dedicated media-understanding plugins.

• Speech-to-text, Transcribe inbound voice messages via batch STT or Voice Call streaming STT providers.

• Media playback, Play assistant audio and video inline in the Control UI and native apps, with managed access and portable playback renditions.

Local media files

Relative paths for view_image, pdf, and reference inputs to image_generate, music_generate, and video_generate resolve against the task's working directory. Even when the agent's default workspace sits somewhere else, a task running in a Git worktree can read media from that worktree.

Workspace-only access is governed by the session's approved filesystem root, which may take in parent directories of the current working directory. The Gateway's media store is also readable by host tools, including browser screenshots and staged inbound attachments. Anything outside these roots, whether a path or a symlink, is rejected. Sandboxed tools read via the sandbox filesystem; picking a host worktree grants no access beyond the sandbox.

Provider capability matrix

Note

The table below lists the dedicated media-generation, TTS, and STT plugins. Inbound media is also understood by many chat-model providers (Anthropic, Google, OpenAI, and others) through their reply model; the full provider list appears in Media understanding.

Note

In this table, Realtime voice refers to provider-native bidirectional realtime (Talk realtime mode, e.g. Gemini Live or the OpenAI Realtime API); today only Google and OpenAI register it. Voice Call streaming STT (one-way audio-to-text) is registered separately by Deepgram, ElevenLabs, Mistral, OpenAI, and xAI; see Speech-to-text and Voice Call below. xAI Realtime voice exists upstream as a capability, but OpenClaw does not register it until the shared realtime-voice contract is able to represent it.

Async vs synchronous

With async tools, OpenClaw hands the request to the provider, returns a task id right away, and tracks the job inside the media runtime. Meanwhile the agent keeps answering other messages. Once the provider is done, OpenClaw wakes the agent with the paths to the generated media so it can inform the user through the session's normal visible-reply mode: automatic final reply delivery when that is configured, or message(action="send") when the session requires the message tool. Should the requester session be inactive, or its active wake fail, and some generated media is still absent from the completion reply, OpenClaw sends a direct fallback carrying only the missing media, with no duplicate sends on retries. Media the completion reply already delivered is not posted a second time.

When completion delivery cannot be confirmed and the original session is still around, OpenClaw attempts to keep the generated media references in that session's history. Completion remains bound to the original session and storage location; a replacement session never gets it. Transient queue admission failures get a bounded retry. If queue admission and history storage both stay unavailable, the completion finishes with a diagnostic and process-local media references; durable recovery is not guaranteed. The local history notice does not resend media to the channel.

Speech-to-text and Voice Call

When configured, Deepgram, DeepInfra, ElevenLabs, Google, Groq, Mistral, OpenAI, OpenRouter, SenseAudio, and xAI can each transcribe inbound audio through the batch tools.media.audio path. Channel plugins that preflight a voice note for mention gating or command parsing mark the transcribed attachment on the inbound context, so the shared media-understanding pass reuses that transcript rather than issuing a second STT call for the same audio.

Voice Call streaming STT providers are also registered by Deepgram, ElevenLabs, Mistral, OpenAI, and xAI, which lets live phone audio be forwarded to the chosen vendor without waiting for a recording to finish.

Talk mode is the recommended choice for live user conversations. Batch audio attachments remain on the media path, while browser realtime, native push-to-talk, telephony, and meeting audio should go through Talk events and the session-scoped catalogs the Gateway returns.

Provider mappings (how vendors split across surfaces)

Google

Surfaces for image, video, music, batch TTS, batch STT, backend realtime voice, and media understanding.

OpenAI

Surfaces for image, video, batch TTS, batch STT, Voice Call streaming STT, backend realtime voice, and memory embedding.

DeepInfra

Surfaces for chat/model routing, image generation/editing, text-to-video, batch TTS, batch STT, image media understanding, and memory embedding. Reranking, classification, object-detection, and other native model types are additionally offered by DeepInfra; OpenClaw has no provider contract for those categories yet, so this plugin does not register them.

xAI

Image, video, search, code-execution, batch TTS, batch STT, and Voice Call streaming STT. xAI Realtime voice exists upstream but is not registered in OpenClaw until the shared realtime-voice contract is able to represent it.

Related

• Image generation
• Video generation
• Music generation
• Text-to-speech
• Media playback
• Media understanding
• Audio nodes
• Talk mode