Talk Mode: Speech Configurations for Native, Web, and Realtime

Talk mode supports five distinct runtime configurations:

• Native macOS/iOS/Android Talk: on-device speech recognition paired with Gateway chat and talk.speak text-to-speech. Apple's speech recognition on macOS and iOS can rely on network services, while Android's behavior depends on which speech service is installed. Nodes broadcast the talk capability and specify the talk.* commands they handle.
• iOS Talk (realtime): WebRTC owned by the client for OpenAI realtime setups that choose webrtc transport or leave transport unspecified, covering both framed and frameless transcript and audio events. Explicit gateway-relay, provider-websocket, and non-OpenAI realtime configurations remain on the Gateway-owned relay; non-realtime configurations fall back to the native speech loop.
• Browser Talk: talk.client.create handles client-owned webrtc/provider-websocket sessions, while talk.session.create manages Gateway-owned gateway-relay sessions. managed-room stays reserved for Gateway handoff and walkie-talkie rooms.
• Android Talk (realtime): Android routes through the Gateway-owned relay for realtime when talk.catalog signals that the realtime group is ready and the selected model passes the Android client gate; client-owned WebRTC sessions are never initiated. The Gateway now accepts gpt-live-* relay sessions, yet Android deliberately keeps those models on native speech recognition, Gateway chat, and talk.speak until the relay path is validated from an actual Android device.
• Transcription-only clients: start with talk.session.create({ mode: "transcription", transport: "gateway-relay", brain: "none" }), then use talk.session.appendAudio and talk.session.close for captions or dictation without an assistant voice reply. One-shot uploaded voice notes continue to use the media understanding audio path.

Native Talk runs as a continuous loop: detect speech, forward the transcript to the model through the active session, await the response, then deliver it through the configured Talk provider (talk.speak).

Choose a Talk voice from chat

Once talk.provider and the corresponding talk.providers.<provider> configuration are in place, /voice status reveals the active provider and voice, /voice list [limit] enumerates the available voices, and /voice set <voiceId|name> persists a provider-scoped choice. Discord exposes the same command natively as /talkvoice.

Status and list commands are read-only. Changing the voice demands message-channel ownership or a Gateway client with operator.admin. Configuration problems, provider lookup failures, unknown-voice errors, and permission denials surface directly in chat. A masked API-key value in /voice status only describes the config; it does not confirm that credentials are usable.

Client-owned realtime Talk typically forwards provider tool calls through talk.client.toolCall rather than invoking chat.send directly. GPT-Live WebRTC sessions delegate over a Gateway-owned sideband, and the Gateway ties each delegation to the browser or Gateway-relay Talk session that initiated it. Backend WebSocket bridges follow the standard relay consult path. While a realtime consult is active, clients may call talk.client.steer or talk.session.steer to label spoken input as status, steer, cancel, or followup; GPT-Live delegations are included here. Accepted steering feeds into the active embedded run; rejected steering returns a reason like no_active_run, not_streaming, or compacting. A newer GPT-Live spoken task also replaces the running delegation.

Thin audio clients can request gateway-control-v1 within talk.client.create.capabilities. OpenAI GA Realtime permits this mode only with a Platform API key. On success, the response includes clientControl: { owner: "gateway" }, a 60-second single-use clientSecret, and the relative offer URL /plugins/openai/realtime/calls. The client submits an audio-only SDP offer to that Gateway route and opens no provider data channel. The Gateway attaches the official OpenAI server sideband and manages tools, transcripts, steering, cancellation, and call cleanup while media flows directly between the client and OpenAI. OAuth-only configurations fail visibly rather than reverting to client-owned control. Existing browser clients omit this capability and retain their current ephemeral-token and WebRTC data-channel flow.

Finalized realtime user and assistant utterances are always appended live to the active agent session, so subsequent chat and voice turns share a single history. Client-owned transports report their finalized transcripts with stable entry ids; Gateway relay and Gateway-controlled WebRTC sessions append the same events server-side. Provider sessions also get the bounded realtime profile context used by Discord voice.

Voice-initiated consult runs demand a fresh, exact spoken confirmation before high-impact operations such as sending messages, controlling nodes, browser or computer actions, service changes, destructive shell commands, or publication. This gate applies to runs started through talk.client.toolCall, the Gateway relay, and GPT-Live sideband delegations. The confirmation covers only the canonical final execution arguments and is used once; if a policy or hook alters the approved action, OpenClaw blocks it until the rewritten action is confirmed. Unrelated concurrent runs stay unaffected. When a call ends, OpenClaw may send a compact Voice call changes digest for mutating tools to the session's last non-WebChat delivery target.

Transcription-only Talk emits the same Talk event envelope as realtime and STT/TTS sessions, but relies on mode: "transcription" and brain: "none". Every Talk session broadcasts events on the talk.event channel; clients subscribe to receive partial and final transcript updates (transcript.delta/transcript.done) plus other session telemetry.

Browser Video Talk works with OpenAI Realtime WebRTC and Google Live provider-WebSocket sessions. OpenAI receives a single bounded JPEG when describe_view requests visual context; no continuous camera track is provided. Google Live gets bounded JPEG frames straight from the browser at up to one frame per second, while describe_view reports the camera-stream state. In both cases, camera frames bypass the Gateway, and stopping Talk releases the camera and microphone tracks.

Behavior (macOS)

• Overlay stays active whenever Talk mode is enabled.
• Listening → Thinking → Speaking phase transitions.
• After a brief pause (silence window), the current transcript is sent.
• Replies appear in WebChat (same as typing).
• Interrupt on speech (default on): if the user speaks while the assistant is talking, playback halts and the interruption timestamp is recorded for the next prompt.

Voice directives in replies

The assistant may prepend a single JSON line to a reply to control voice:

{ "voice": "<voice-id>", "once": true }

Rules:

• Only the first non-empty line counts; the JSON line is removed before TTS playback.
• Unknown keys are ignored.
• once: true applies to the current reply only; without it, the voice becomes the new Talk mode default.

Supported keys: voice / voice_id / voiceId, model / model_id / modelId, speed, rate (WPM), stability, similarity, style, speakerBoost, seed, normalize, lang, output_format, latency_tier, once.

Config (~/.openclaw/openclaw.json)

{
  talk: {
    provider: "elevenlabs",
    providers: {
      elevenlabs: {
        voiceId: "elevenlabs_voice_id",
        modelId: "eleven_v3",
        outputFormat: "mp3_44100_128",
        apiKey: "elevenlabs_api_key",
      },
      mlx: {
        modelId: "mlx-community/Soprano-80M-bf16",
        // Fish S2 Pro can also use a local reference voice:
        // referenceAudioPath: "/Users/example/Voices/reference.wav",
        // referenceText: "Exact transcript of the reference clip.",
      },
      system: {},
    },
    speechLocale: "ru-RU",
    silenceTimeoutMs: 1500,
    interruptOnSpeech: true,
    realtime: {
      provider: "openai",
      providers: {
        openai: {
          apiKey: "openai_api_key",
          model: "gpt-realtime-2.1",
          speakerVoice: "cedar",
        },
      },
      instructions: "Speak warmly and keep answers brief.",
      mode: "realtime",
      transport: "webrtc",
      brain: "agent-consult",
    },
  },
}

Native GPT-Live is available through https://api.openai.com/v1/live on both the OpenAI browser WebRTC path and the Gateway-relay Talk route. Assign talk.realtime.model a value of either gpt-live-1-codex (the suggested choice) or gpt-live-1-boulder-alpha; using gpt-live-1 or gpt-live-1-mini here will not work. For browser and Gateway-relay WebRTC, a ChatGPT OAuth subscription profile is tried first, with Platform API-key auth as the fallback. All other backend bridges attach directly over the Frameless Bidi WebSocket and demand Platform API-key auth, and its /v1/live access is currently restricted to a waitlist.

The Control UI offers the fastest path to configuration: go to Settings → Talk, select OpenAI, and choose a gpt-live-* model. Before that, an OpenClaw auth profile created with openclaw models auth login --provider openai must exist to satisfy the OAuth requirement; a prior Codex CLI sign-in will not be detected. GPT-Live additionally needs the bundled openai plugin active in full mode; if the plugins.allow list is too restrictive, session creation fails with "OpenAI GPT-Live browser session broker is unavailable". Operational limits include 8 concurrent sessions per Gateway and a session TTL of 30 minutes. Browser sessions also rely on 60-second single-use offer tokens.

GPT-Live recognizes alloy, ash, ballad, cedar, coral, echo, marin, sage, shimmer, and verse. A 403 Voice session access denied reply carries a dual meaning: an invalid voice triggers the same response. The older chatgpt.com backend route likewise returns 403; OpenClaw opts for the native api.openai.com/v1/live route instead.

Relay clients never see OAuth or Platform credentials when the Gateway-owned WebRTC route is used. For backend WebSocket paths, the Platform key remains on the Gateway; OpenClaw handles conversion between telephony G.711 u-law audio and GPT-Live's 24 kHz PCM format in both directions.

For GA gpt-realtime-2.1, gpt-realtime-2.1-mini, and gpt-realtime-2 browser sessions, Platform credentials take precedence in this sequence: the configured realtime API key, an openai API-key profile, and finally OPENAI_API_KEY. When none are set up, browser Talk switches to an OpenClaw ChatGPT OAuth profile and passes SDP through the Gateway's single-use offer broker, keeping the OAuth token out of the browser. If a configured Platform credential cannot be resolved, the connection fails closed rather than quietly dropping back to OAuth.

Platform-key-only access still applies to iOS client-owned WebRTC, Voice Call, GA Gateway relay, provider WebSocket transports, Discord realtime voice, and Android realtime. GA browser Talk retains its existing client-owned data channel and talk.client.toolCall loop; under OAuth, only the credential owner and the SDP exchange path differ. GPT-Live Gateway relay starts with ChatGPT OAuth and falls back to waitlist-enabled Platform access.

talk.catalog provides canonical provider ids and registry aliases, along with each provider's valid modes, transports, brain strategies, realtime audio formats, capability flags, and the readiness result chosen at runtime. First-party Talk clients should consult that catalog rather than keeping provider aliases locally; treat an older Gateway that lacks group readiness as unverified, not as definitively unconfigured. Streaming transcription providers are located via talk.catalog.transcription; the current Gateway relay uses the Voice Call streaming provider config until a dedicated Talk transcription config surface becomes available.

macOS UI

• Menu bar toggle: Talk
• Config tab: Talk Mode group (voice id plus interrupt toggle)
• Overlay: the orb displays the universal talk waveform (shared with iOS, watchOS, and Android). Listening follows the live mic level, Speaking follows the actual TTS playback envelope, Thinking breathes softly. Click the orb to pause or resume, double-click to stop speaking, click X to exit Talk mode.

Android UI

• Android's main navigation is Home, Chat, and Settings. Voice input is in the Chat composer, not a separate Voice tab.
• Tap the composer microphone for on-device dictation. Long-press it to record a voice-note attachment. Start continuous Talk from the Talk waveform.
• Dictation, voice-note recording, and Talk are mutually exclusive microphone paths; starting one stops or blocks the others.
• Realtime Talk prefers a connected Bluetooth Classic or BLE headset microphone; if it disconnects, the app requests another headset input or falls back to the default microphone, restoring the default preference once capture stops.
• Dictation and voice-note recording stop when the app leaves the foreground or the user leaves Chat.
• Talk Mode keeps running until toggled off or the node disconnects, using Android's microphone foreground-service type while active.
• Android supports pcm_16000, pcm_22050, pcm_24000, and pcm_44100 output formats for low-latency AudioTrack streaming.

Notes

• Requires Speech + Microphone permissions.
• Native Talk uses the active Gateway session and only falls back to history polling when response events are unavailable.
• The gateway resolves Talk playback through talk.speak using the active Talk provider. Android falls back to local system TTS only when that RPC is unavailable.
• macOS local MLX playback uses the bundled openclaw-mlx-tts helper when present, or an executable on PATH. Set OPENCLAW_MLX_TTS_BIN to point at a custom helper binary during development. The helper streams PCM, keeps one selected model resident, and supports Fish S2 Pro reference audio through providers.mlx.referenceAudioPath plus referenceText.
• Voice directive value ranges (ElevenLabs): stability, similarity, and style accept 0..1; speed accepts 0.5..2; latency_tier accepts 0..4.

Related

• Voice wake
• Audio and voice notes
• Media understanding