Video Generation with OpenClaw: Text, Image, and Video Inputs

OpenClaw agents can produce videos from text prompts, reference images, or existing videos via video_generate. The system supports sixteen provider backends, and the agent automatically selects the appropriate one based on configuration and the API keys that are available.

Note

video_generate shows up only when at least one video-generation provider is present. If you don't see it among your agent tools, either provide a provider API key or set up agents.defaults.mediaModels.video.

Three runtime modes are available in video_generate, and which one applies is determined by the reference inputs passed in the call:

• generate - no reference media is supplied (text-to-video).
• imageToVideo - one or more reference images are supplied.
• videoToVideo - one or more reference videos are supplied.

A provider might support any combination of these modes. Before submission, the tool checks that the active mode is valid, and the supported modes are listed in action=list.

Quick start

Configure auth

For any provider you want to use, configure an API key:

export GEMINI_API_KEY="your-key"

Pick a default model (optional)

openclaw config set agents.defaults.mediaModels.video.primary "google/veo-3.1-fast-generate-preview"

Ask the agent

Create a 5-second cinematic clip of a friendly lobster surfing at sunset.

video_generate gets called by the agent automatically. There's no need to allowlist any tools.

How async generation works

Video generation runs asynchronously:

1. OpenClaw sends the request to the provider and gets back a task id right away.
2. The provider handles the job in the background (usually 30 seconds to a few minutes, depending on the provider and resolution; providers backed by slow queues may run until the configured timeout).
3. Once the video is done, OpenClaw wakes the same session using an internal completion event.
4. The agent reports it through the session's standard visible-reply mechanism: an automatic final reply, or message(action="send") if the session needs the message tool. If the requesting session is not active, or if waking it fails and the generated media is still absent from the completion reply, OpenClaw delivers an idempotent direct fallback that includes the media.

While a job is still running, repeated video_generate calls within the same session just return the current task status rather than kicking off another generation. To check without starting anything new, use action: "status", or from the CLI use openclaw tasks list / openclaw tasks show <lookup> (see Background tasks).

For runs that aren't backed by a session (like direct tool calls), the tool switches to inline generation and returns the final media path in the same turn.

When the provider returns bytes, the generated video files are stored in OpenClaw-managed media storage. The default limit is 16MB (the shared video media cap); agents.defaults.mediaMaxMb increases that for larger renders. If a provider also gives back a hosted output URL, OpenClaw hands over that URL instead of failing the task when local persistence rejects an oversized file.

Task lifecycle

You can check status from the CLI:

openclaw tasks list
openclaw tasks show <lookup>
openclaw tasks cancel <lookup>

Supported providers

Certain providers also support extra or alternative API key environment variables. For specifics, check the individual provider pages.

To see which providers, models, and runtime modes are available at runtime, execute video_generate action=list.

Capability matrix

This is the explicit mode contract that video_generate, contract tests, and the shared live sweep rely on.

Tool parameters

Required

• prompt (string, required), Provide a text prompt describing the video you want to generate. This is mandatory for action: "generate".

Content inputs

• image (string), One reference image, given as a path or URL.

• images (string[]), Several reference images, with a maximum of 9.

• imageRoles (string[]), Role hints that are optional and align by position with the full image list. Accepted values: first_frame, last_frame, reference_image.

• video (string), One reference video, given as a path or URL.

• videos (string[]), Several reference videos, with a maximum of 4.

• videoRoles (string[]), Role hints that are optional and align by position with the full video list. Accepted value: reference_video.

• audioRef (string), One reference audio, given as a path or URL. This serves as background music or a voice reference when the provider accepts audio inputs.

• audioRefs (string[]), Several reference audios, with a maximum of 3.

• audioRoles (string[]), Role hints that are optional and align by position with the full audio list. Accepted value: reference_audio.

Note

Role hints travel to the provider untouched. The canonical set is defined by the VideoGenerationAssetRole union, though some providers recognize extra role strings. *Roles arrays must stay within the length of the matching reference list; getting the count wrong by one triggers an explicit error. An empty string is the way to leave a slot blank. With xAI, assign every image role the value reference_image to activate its reference_images generation mode; for single-image image-to-video, either drop the role or set it to first_frame.

Style controls

• aspectRatio (string), Aspect-ratio hint, for instance 1:1, 16:9, 9:16, adaptive, or something provider-specific. Depending on the provider, OpenClaw either normalizes or discards values it does not recognize.

• resolution (string), Resolution hint, for instance 360P, 480P, 540P, 720P, 768P, 1080P, 4K, or something provider-specific. Depending on the provider, OpenClaw either normalizes or discards values it does not recognize.

• durationSeconds (number), Desired length in seconds, rounded to the nearest value the provider supports.

• size (string), Size hint, applicable when the provider supports it.

• audio (boolean), Turn on generated audio in the output where supported. This differs from audioRef*, which concerns inputs.

• watermark (boolean), Switch provider watermarking on or off where supported.

adaptive acts as a provider-specific sentinel: it passes through unchanged to providers that list adaptive in their capabilities (BytePlus Seedance, for example, uses it to infer the ratio automatically from the dimensions of the input image). For providers that do not list it, the value surfaces through details.ignoredOverrides in the tool result, making the drop observable.

Advanced

• action (generate" | "status" | "list, default: generate), "status" fetches the active session task; "list" looks at providers.

• model (string), Override for provider or model, such as runway/gen4.5.

• filename (string), Hint for the output filename.

• timeoutMs (number), Optional timeout in milliseconds for the provider operation. When left out, OpenClaw falls back to agents.defaults.mediaModels.video.timeoutMs if that is set, otherwise to the plugin-defined provider default when one exists.

• providerOptions (object), Provider-specific options delivered as a JSON object, for example {"seed": 42, "draft": true}. Providers that expose a typed schema check both keys and types; keys that are unknown or types that do not match cause the candidate to be skipped during fallback. Providers without a declared schema take the options as-is. Run video_generate action=list to see what each provider accepts.

Note

Parameters are not universally supported across providers. OpenClaw rounds duration to the nearest supported value and converts translated geometry hints, such as size-to-aspect-ratio, when a fallback provider offers a different control surface. Overrides that are genuinely unsupported get ignored on a best-effort basis and show up as warnings in the tool result. Hard capability limits, like exceeding the reference input count, fail before submission. The tool result reports what was applied, and details.normalization records any translation from what was requested to what was applied.

The runtime mode is chosen by the reference media:

• No reference media -> generate
• Any image reference -> imageToVideo
• Any video reference -> videoToVideo
• Reference audio inputs do not alter the resolved mode; they layer on top of whichever mode the image/video references pick, and they only work with providers that declare maxInputAudios.

Mixing image and video references is not a reliable shared capability surface. Stick to one reference type per request.

Fallback and typed options

Certain capability checks happen at the fallback layer rather than at the tool boundary, so a request that goes beyond the primary provider's limits can still succeed on a capable fallback:

• An active candidate that declares no maxInputAudios (or 0) is passed over when the request carries audio references, and the next candidate is tried. The same guard applies to image and video reference counts against maxInputImages/maxInputVideos.
• An active candidate whose maxDurationSeconds sits below the requested durationSeconds with no declared supportedDurationSeconds list -> skipped.
• The request contains providerOptions and the active candidate explicitly declares a typed providerOptions schema -> skipped if the supplied keys are absent from the schema or the value types do not line up. Providers without a declared schema receive options as-is (backward-compatible pass-through). A provider can refuse all provider options by declaring an empty schema (capabilities.providerOptions: {}), which triggers the same skip as a type mismatch.

The first skip reason in a request logs at warn so operators can see when their primary provider was bypassed; later skips log at debug to keep long fallback chains quiet. When every candidate is skipped, the aggregated error lists the skip reason for each one.

Actions

Model selection

The model is picked by OpenClaw using this priority:

1. model tool parameter - when the agent passes one in the call.
2. videoGenerationModel.primary taken from config.
3. videoGenerationModel.fallbacks in sequence.
4. Auto-detection - providers with valid auth are tried, starting from the current default provider and moving through the rest alphabetically.

A failed provider causes the system to try the next option on its own. When every candidate has failed, the returned error carries details from each attempt.

Fallback across authenticated providers is on by default. A per-call model still takes precedence.

{
  agents: {
    defaults: {
      videoGenerationModel: {
        primary: "google/veo-3.1-fast-generate-preview",
        fallbacks: ["runway/gen4.5", "qwen/wan2.6-t2v"],
        timeoutMs: 180000, // optional per-tool provider request timeout override
      },
    },
  },
}

Provider notes

Alibaba

Relies on the DashScope / Model Studio async endpoint. Reference images and videos have to be hosted at remote http(s) URLs.

BytePlus plugin

Needs the official @openclaw/byteplus-provider plugin. Provider id: byteplus.

Models: seedance-1-0-pro-250528 (default), seedance-1-5-pro-251215.

Built on the unified content[] API. Accepts up to 2 input images (first_frame + last_frame). Provide images by position, or give each image an explicit role.

Recognized providerOptions keys: seed (number), draft (boolean - forces 480p), camera_fixed (boolean).

BytePlus Seedance 1.5 plugin

Depends on the @openclaw/byteplus-modelark plugin (external, not bundled). Provider id: byteplus-seedance15. Model: seedance-1-5-pro-251215.

Built on the unified content[] API. Handles at most 2 input images (first_frame + last_frame). Every input must be a remote https:// URL. Assign role: "first_frame" / "last_frame" to each image, or supply images by position.

aspectRatio: "adaptive" derives the aspect ratio from the input image on its own. audio: true is translated to generate_audio. providerOptions.seed (number) gets passed through.

BytePlus Seedance 2.0

Requires the @openclaw/byteplus-modelark plugin (external, not bundled). Provider id: byteplus-seedance2. Models: dreamina-seedance-2-0-260128, dreamina-seedance-2-0-fast-260128.

Built on the unified content[] API. Supports up to 9 reference images, 3 reference videos, and 3 reference audios. All inputs must be remote https:// URLs. Set role on each asset - accepted values: "first_frame", "last_frame", "reference_image", "reference_video", "reference_audio".

aspectRatio: "adaptive" derives the aspect ratio from the input image on its own. audio: true is translated to generate_audio. providerOptions.seed (number) gets passed through.

ComfyUI

Runs locally or in the cloud, driven by a workflow. Handles text-to-video and image-to-video through the graph you configure.

fal

Uses a queue-backed flow for long-running jobs. Before treating an in-progress fal queue job as timed out, OpenClaw waits up to 20 minutes by default. Most fal video models accept a single image reference. Seedance 2.0 reference-to-video models accept up to 9 images, 3 videos, and 3 audio references, with at most 12 total reference files.

Google (Gemini / Veo)

One image or one video reference is supported. On the Gemini API path, generated-audio requests are ignored with a warning because that API rejects the generateAudio parameter for current Veo video generation.

MiniMax

Only a single image reference is allowed. MiniMax accepts 768P and 1080P resolutions; requests such as 720P are normalized to the closest supported value before submission.

OpenAI

Only the size override is forwarded. Other style overrides (aspectRatio, resolution, audio, watermark) are ignored with a warning.

OpenRouter

OpenRouter's asynchronous /videos API is used. OpenClaw submits the job, polls polling_url, and downloads either unsigned_urls or the documented job content endpoint. The bundled google/veo-3.1-fast default advertises 4/6/8 second durations, 720P/1080P resolutions, and 16:9/9:16 aspect ratios.

Qwen

Same DashScope backend as Alibaba. Reference inputs must be remote http(s) URLs; local files are rejected upfront.

Runway

Local files are supported via data URIs. Video-to-video requires runway/gen4_aleph. Text-only runs expose 16:9 and 9:16 aspect ratios.

Together

Only a single image reference is supported.

Vydra

https://www.vydra.ai/api/v1 is used directly to avoid auth-dropping redirects. veo3 is text-to-video only; kling requires a remote image URL.

xAI

The default grok-imagine-video model supports text-to-video, single first-frame image-to-video, up to 7 reference_image inputs through xAI reference_images, and remote video edit/extend flows. Generation defaults to 480P; single-image image-to-video inherits the source ratio when aspectRatio is omitted. Video edit/extend inherit the input geometry and do not accept aspect-ratio or resolution overrides. Extension accepts 2-10 seconds.

grok-imagine-video-1.5 is image-to-video only: provide exactly one image. It supports 1-15 seconds and 480P, 720P, or 1080P, defaulting to 480P; omit aspectRatio to inherit the source image ratio. The preview and dated 1.5 identifiers receive the same validation and are forwarded unchanged.

Provider capability modes

The shared video-generation contract supports mode-specific capabilities instead of only flat aggregate limits. New provider implementations should prefer explicit mode blocks:

capabilities: {
  generate: {
    maxVideos: 1,
    maxDurationSeconds: 10,
    supportsResolution: true,
  },
  imageToVideo: {
    enabled: true,
    maxVideos: 1,
    maxInputImages: 1,
    maxInputImagesByModel: { "provider/reference-to-video": 9 },
    maxDurationSeconds: 5,
  },
  videoToVideo: {
    enabled: true,
    maxVideos: 1,
    maxInputVideos: 1,
    maxDurationSeconds: 5,
  },
}

Flat aggregate fields such as maxInputImages and maxInputVideos are not enough to advertise transform-mode support. Providers should declare generate, imageToVideo, and videoToVideo explicitly so live tests, contract tests, and the shared video_generate tool can validate mode support deterministically.

When one model in a provider has wider reference-input support than the rest, use maxInputImagesByModel, maxInputVideosByModel, or maxInputAudiosByModel instead of raising the mode-wide limit.

Live tests

Opt-in live coverage for the shared bundled providers:

OPENCLAW_LIVE_TEST=1 pnpm test:live -- extensions/video-generation-providers.live.test.ts

Repo wrapper:

pnpm test:live:media video

This live file uses already-exported provider env vars ahead of stored auth profiles by default, and runs a release-safe smoke by default.

• generate for each provider in the sweep that is not FAL.
• A one-second lobster prompt.
• The per-provider operation limit, sourced from OPENCLAW_LIVE_VIDEO_GENERATION_TIMEOUT_MS (defaulting to 180000).

FAL is not enabled by default, since queue latency on the provider side can end up dominating the release time:

pnpm test:live:media video --video-providers fal

To also run the declared transform modes that the shared sweep can safely handle with local media, set OPENCLAW_LIVE_VIDEO_GENERATION_FULL_MODES=1:

• imageToVideo when capabilities.imageToVideo.enabled.
• videoToVideo when capabilities.videoToVideo.enabled and the provider/model supports buffer-backed local video input in the shared sweep.

The shared videoToVideo live lane currently covers runway only when you pick runway/gen4_aleph.

Configuration

To set the default video-generation model, update your OpenClaw config:

{
  agents: {
    defaults: {
      videoGenerationModel: {
        primary: "qwen/wan2.6-t2v",
        fallbacks: ["qwen/wan2.6-r2v-flash"],
      },
    },
  },
}

Alternatively, use the CLI:

openclaw config set agents.defaults.mediaModels.video.primary "qwen/wan2.6-t2v"

Related

• Alibaba Model Studio
• Background tasks - task tracking for async video generation
• BytePlus
• ComfyUI
• Configuration reference
• fal
• Google (Gemini)
• MiniMax
• Models
• OpenAI
• Qwen
• Runway
• Together AI
• Tools overview
• Vydra
• xAI