Music Generation with music_generate: Providers and Workflows

The music_generate tool generates music or audio through the shared music-generation capability, which is powered by ComfyUI, fal, Google, MiniMax, and OpenRouter.

Note

music_generate shows up only when at least one music-generation provider is present: either an explicit agents.defaults.mediaModels.music configuration or a provider set up through authentication, such as a configured API key.

In session-backed agent runs, music_generate launches as a background task, logs its progress in the task ledger, and then notifies the agent once the track finishes so it can inform the user and attach the resulting audio. The completion agent adheres to the session's visible-reply contract: it sends an automatic final reply when configured, or message(action="send") when the session requires the message tool. If the requesting session is not active, or if its wake fails and the generated audio is still absent from the reply, OpenClaw delivers an idempotent direct fallback containing only the missing audio.

Quick start

Shared provider-backed

Configure auth

Configure an API key for at least one provider, for instance GEMINI_API_KEY or MINIMAX_API_KEY.

Pick a default model (optional)

{
  agents: {
    defaults: {
      musicGenerationModel: {
        primary: "google/lyria-3-clip-preview",
      },
    },
  },
}

Ask the agent

"Generate an upbeat synthpop track about a night drive through a neon city."

The agent invokes music_generate on its own. No tool allow-listing is required.

When there is no session-backed agent run, such as in direct or local contexts, the tool executes inline and returns the final media path within the same tool result.

ComfyUI workflow

Configure the workflow

Set up plugins.entries.comfy.config.music with a workflow JSON and nodes for prompts and outputs.

Cloud auth (optional)

For Comfy Cloud, specify COMFY_API_KEY or COMFY_CLOUD_API_KEY.

Call the tool

/tool music_generate prompt="Warm ambient synth loop with soft tape texture"

Example prompts:

Generate a cinematic piano track with soft strings and no vocals.

Generate an energetic chiptune loop about launching a rocket at sunrise.

Use action: "list" to see which providers and models are available, and action: "status" to check the active session-backed music task:

/tool music_generate action=list
/tool music_generate action=status

Direct generation example:

/tool music_generate prompt="Dreamy lo-fi hip hop with vinyl texture and gentle rain" instrumental=true

Supported providers

MiniMax registers two provider ids that share the same models: minimax for API-key authentication and minimax-portal for OAuth. Model references follow the auth path (minimax/music-2.6 versus minimax-portal/music-2.6); consult MiniMax.

fal offers fal-ai/ace-step/prompt-to-audio (wav, no lyrics, no instrumental toggle) and fal-ai/stable-audio-25/text-to-audio (wav, prompt-only) in addition to its default MiniMax-backed model. Google's default lyria-3-clip-preview produces only mp3; lyria-3-pro-preview also supports wav. MiniMax additionally provides music-2.6-free, music-cover, and music-cover-free. OpenRouter also provides google/lyria-3-clip-preview.

Capability matrix

The explicit mode contract used by music_generate, contract tests, and the shared live sweep:

Tool parameters

• prompt (string, required), Prompt for music generation. Needed for action: "generate".

• action (generate" | "status" | "list, default: generate), "status" returns the current session task; "list" inspects providers.

• model (string), Override for provider/model (e.g. google/lyria-3-pro-preview, comfy/workflow).

• lyrics (string), Optional lyrics when the provider accepts explicit lyric input.

• instrumental (boolean), Ask for instrumental-only output when the provider supports it.

• image (string), Single reference image path or URL.

• images (string[]), Multiple reference images (up to 10 on supporting providers).

• durationSeconds (number), Target duration in seconds when the provider supports duration hints.

• format (mp3" | "wav), Output format hint when the provider supports it.

• filename (string), Output filename hint.

Note

Not all providers support all parameters. OpenClaw still validates hard limits such as input counts before submission. When a provider supports duration but uses a shorter maximum than the requested value, OpenClaw clamps to the closest supported duration. Truly unsupported optional hints are ignored with a warning when the selected provider or model cannot honor them. Tool results report applied settings; details.normalization captures any requested-to-applied mapping.

Provider request timeouts are operator configuration only. OpenClaw uses agents.defaults.mediaModels.music.timeoutMs when configured, raises values below 120000ms to 120000ms, and otherwise defaults provider requests to 300000ms.

Async behavior

Session-backed music generation runs as a background task:

• Background task: music_generate creates a background task, returns a started/task response immediately, and posts the finished track later in a follow-up agent message.
• Duplicate prevention: while a task is queued or running, later music_generate calls in the same session return task status instead of starting another generation. Use action: "status" to check explicitly. A recently completed matching request is also deduplicated for 2 minutes.
• Status lookup: openclaw tasks list or openclaw tasks show <taskId> inspects queued, running, and terminal status.
• Completion wake: OpenClaw injects an internal completion event back into the same session so the model can write the user-facing follow-up itself.
• Prompt hint: later user/manual turns in the same session get a small runtime hint when a music task is already in flight, so the model does not blindly call music_generate again.
• No-session fallback: direct/local contexts without a real agent session run inline and return the final audio result in the same turn.

Task lifecycle

The music task surfaces the same states as the general task registry (see Background tasks for the full state machine, including timed_out, cancelled, and lost). Most music runs move through:

Check status from the CLI:

openclaw tasks list
openclaw tasks show <taskId>
openclaw tasks cancel <taskId>

Configuration

Model selection

{
  agents: {
    defaults: {
      musicGenerationModel: {
        primary: "google/lyria-3-clip-preview",
        fallbacks: ["fal/fal-ai/minimax-music/v2.6", "minimax/music-2.6"],
      },
    },
  },
}

Provider selection order

OpenClaw attempts providers in the following sequence:

1. The model parameter taken from the tool call, provided the agent supplies one.
2. musicGenerationModel.primary sourced from configuration.
3. musicGenerationModel.fallbacks, evaluated sequentially.
4. Auto-detection that relies exclusively on auth-backed provider defaults:
   • the current default text-model provider comes first, assuming it also supports music generation;
   • the rest of the registered music-generation providers follow, ordered alphabetically by provider id.

When one provider fails, the system moves on to the next candidate automatically. If every provider fails, the resulting error message carries details gathered from each individual attempt.

Automatic failover among authenticated providers is permanently active. A per-call model continues to hold final authority.

Provider notes

ComfyUI

This path is workflow-driven, depending on the configured graph and how prompt/output fields are mapped to nodes. The comfy plugin connects to the shared music_generate tool via the music-generation provider registry.

fal

fal model endpoints are used through the common provider auth path. The provider bundled by default points to fal-ai/minimax-music/v2.6 and additionally offers fal-ai/ace-step/prompt-to-audio and fal-ai/stable-audio-25/text-to-audio for prompt-to-audio requests. Instrumental mode and lyrics are restricted to MiniMax models; the other two models accept prompts only.

Google (Lyria 3)

Lyria 3 batch generation is what this provider uses. The bundled flow as it stands supports a prompt, optional lyrics text, and optional reference images. The default lyria-3-clip-preview model produces mp3 exclusively, while the lyria-3-pro-preview model also handles wav.

MiniMax

The batch music_generation endpoint is what this one relies on. It accepts a prompt, optional lyrics, instrumental mode, and mp3 output, authenticated either through minimax API-key credentials or minimax-portal OAuth. Models music-2.6-free, music-cover, and music-cover-free are also made available.

OpenRouter

OpenRouter chat completions with streaming enabled supply the audio output here. The bundled provider defaults to google/lyria-3-pro-preview and also exposes openrouter/google/lyria-3-clip-preview.

Choosing the right path

• Choose the shared provider-backed route when model selection, provider failover, and the built-in async task/status flow are what you need.
• Choose the plugin path (ComfyUI) when a custom workflow graph or a provider outside the shared bundled music capability is required.

For ComfyUI-specific debugging, refer to ComfyUI. For shared provider debugging, begin with fal, Google (Gemini), MiniMax, or OpenRouter.

Provider capability modes

Explicit mode declarations are part of the shared music-generation contract:

• generate covers generation from a prompt alone.
• edit applies when the request carries one or more reference images.

New provider implementations are advised to rely on explicit mode blocks:

capabilities: {
  generate: {
    maxTracks: 1,
    supportsLyrics: true,
    supportsFormat: true,
  },
  edit: {
    enabled: true,
    maxTracks: 1,
    maxInputImages: 1,
    supportsFormat: true,
  },
}

Legacy flat fields like maxInputImages, supportsLyrics, and supportsFormat are not sufficient to signal edit support. Providers must declare generate and edit explicitly so that live tests, contract tests, and the shared music_generate tool can verify mode support in a deterministic way.

Live tests

Opt-in live coverage for the shared bundled providers (fal, Google, MiniMax, OpenRouter):

OPENCLAW_LIVE_TEST=1 pnpm test:live -- extensions/music-generation-providers.live.test.ts

The equivalent repo wrapper, which runs the same test file:

pnpm test:live:media:music

By default this live file prioritizes already-exported provider env vars over stored auth profiles, and it executes both generate and declared edit coverage when the provider has edit mode enabled. Current coverage:

• google: generate plus edit
• fal: generate only
• minimax: generate only
• openrouter: generate plus edit
• comfy: separate Comfy live coverage, not part of the shared provider sweep

Opt-in live coverage for the bundled ComfyUI music path:

OPENCLAW_LIVE_TEST=1 COMFY_LIVE_TEST=1 pnpm test:live -- extensions/comfy/comfy.live.test.ts

When those sections are configured, the Comfy live file also covers comfy image and video workflows.

Related

• Background tasks, monitoring for music_generate processes running in the background
• ComfyUI
• Configuration reference, settings for musicGenerationModel
• Google (Gemini)
• MiniMax
• Models, setup and fallback options for models
• Tools overview