Talk Mode: Continuous Speech Conversations Across STT/TTS and Realtime Voice

Talk mode supports five runtime configurations:

• Native macOS/iOS/Android Talk: Uses built-in speech recognition, Gateway chat, and talk.speak for text-to-speech. Apple Speech on macOS/iOS may rely on network services; Android's behavior depends on the speech service installed. Nodes advertise the talk capability and list the talk.* commands they support.
• iOS Talk (realtime): Client-owned WebRTC is used for OpenAI realtime configurations that specify webrtc transport or leave transport unspecified. Explicit gateway-relay, provider-websocket, and non-OpenAI realtime configurations continue using the Gateway-owned relay; non-realtime configurations follow the native speech loop.
• Browser Talk: Uses talk.client.create for client-owned webrtc/provider-websocket sessions, or talk.session.create for Gateway-owned gateway-relay sessions. managed-room is set aside for Gateway handoff and walkie-talkie rooms.
• Android Talk (realtime): Requires opting in with talk.realtime.mode: "realtime" and talk.realtime.transport: "gateway-relay". Otherwise, Android uses native speech recognition, Gateway chat, and talk.speak.
• Transcription-only clients: Start with talk.session.create({ mode: "transcription", transport: "gateway-relay", brain: "none" }), then use talk.session.appendAudio, talk.session.cancelTurn, and talk.session.close for captions or dictation without an assistant voice response. One-shot uploaded voice notes still go through the media understanding audio path.

Native Talk operates as a continuous loop: it listens for speech, sends the transcript to the model through the active session, waits for a response, and then speaks it using the configured Talk provider (talk.speak).

Client-owned realtime Talk forwards provider tool calls through talk.client.toolCall instead of calling chat.send directly. While a realtime consult is active, clients can call talk.client.steer or talk.session.steer to classify spoken input as status, steer, cancel, or followup. Accepted steering queues into the active embedded run; rejected steering returns a reason like no_active_run, not_streaming, or compacting.

Finalized realtime user and assistant utterances are always appended live to the active agent session, so later chat and voice turns share a single history. Client-owned transports report their finalized transcripts with stable entry IDs; Gateway relay sessions append the same events server-side. Provider sessions also receive the bounded realtime profile context used by Discord voice.

Voice-originated consult runs require a new, exact spoken confirmation before high-impact actions like sending messages, controlling nodes, browser or computer actions, service changes, destructive shell commands, or publication. The confirmation applies only to the exact blocked tool arguments and is consumed once; unrelated concurrent runs remain unaffected. When a call closes, OpenClaw can send a compact Voice call changes digest for mutating tools to the session's last non-WebChat delivery target.

Transcription-only Talk emits the same Talk event envelope as realtime and STT/TTS sessions, but uses mode: "transcription" and brain: "none". All Talk sessions broadcast events on the talk.event channel; clients subscribe to it for partial and final transcript updates (transcript.delta/transcript.done) and other session telemetry.

Browser Video Talk is available for OpenAI Realtime WebRTC and Google Live provider-WebSocket sessions. OpenAI receives a single bounded JPEG when describe_view requests visual context; it does not get a continuous camera track. Google Live receives bounded JPEG frames directly from the browser at up to one frame per second, while describe_view reports the camera-stream state. In both cases, camera frames bypass the Gateway, and stopping Talk releases the camera and microphone tracks.

Behavior (macOS)

• Overlay is always visible while Talk mode is active.
• Phase transitions: Listening → Thinking → Speaking.
• After a short pause (silence window), the current transcript is sent.
• Replies appear in WebChat (as if typed).
• Interrupt on speech (enabled by default): if the user speaks while the assistant is talking, playback stops and the interruption timestamp is recorded for the next prompt.

Voice directives in replies

The assistant can prefix a reply with a single JSON line to control voice:

{ "voice": "<voice-id>", "once": true }

Rules:

• Only the first non-empty line is considered; the JSON line is removed before TTS playback.
• Unknown keys are ignored.
• once: true applies only to the current reply; without it, the voice becomes the new default for Talk mode.

The following keys are supported: voice, voice_id, voiceId, model, model_id, modelId, speed, rate (WPM), stability, similarity, style, speakerBoost, seed, normalize, lang, output_format, latency_tier, and once.

Config (~/.openclaw/openclaw.json)

{
  talk: {
    provider: "elevenlabs",
    providers: {
      elevenlabs: {
        voiceId: "elevenlabs_voice_id",
        modelId: "eleven_v3",
        outputFormat: "mp3_44100_128",
        apiKey: "elevenlabs_api_key",
      },
      mlx: {
        modelId: "mlx-community/Soprano-80M-bf16",
      },
      system: {},
    },
    speechLocale: "ru-RU",
    silenceTimeoutMs: 1500,
    interruptOnSpeech: true,
    realtime: {
      provider: "openai",
      providers: {
        openai: {
          apiKey: "openai_api_key",
          model: "gpt-realtime-2.1",
          speakerVoice: "cedar",
        },
      },
      instructions: "Speak warmly and keep answers brief.",
      mode: "realtime",
      transport: "webrtc",
      brain: "agent-consult",
    },
  },
}

talk.catalog reveals canonical provider identifiers and registry aliases, along with each provider's supported modes, transports, brain strategies, realtime audio formats, capability flags, and the runtime-selected readiness outcome. First-party Talk clients should consult this catalog instead of keeping provider aliases locally; treat an older Gateway that lacks group readiness as unverified rather than definitively unconfigured. Streaming transcription providers are discovered via talk.catalog.transcription; the current Gateway relay uses the Voice Call streaming provider configuration until a dedicated Talk transcription configuration surface ships.

macOS UI

• Menu bar toggle: Talk
• Config tab: Talk Mode group (voice id plus interrupt toggle)
• Overlay: the orb shows the universal talk waveform, shared across iOS, watchOS, and Android. Listening tracks the live microphone level, Speaking follows the actual TTS playback envelope, and Thinking breathes softly. Click the orb to pause or resume, double-click to stop speaking, and click X to exit Talk mode.

Android UI

• Android's primary navigation consists of Home, Chat, and Settings. Voice input appears in the Chat composer rather than a separate Voice tab.
• Tap the composer microphone for on-device dictation. Long-press it to record a voice-note attachment. Start continuous Talk from the Talk waveform.
• Dictation, voice-note recording, and Talk are mutually exclusive microphone paths; starting one stops or blocks the others.
• Realtime Talk prefers a connected Bluetooth Classic or BLE headset microphone; if that disconnects, the app requests another headset input or falls back to the default microphone, restoring the default preference once capture stops.
• Dictation and voice-note recording stop when the app leaves the foreground or the user leaves Chat.
• Talk Mode continues running until toggled off or the node disconnects, using Android's microphone foreground-service type while active.
• Android supports pcm_16000, pcm_22050, pcm_24000, and pcm_44100 output formats for low-latency AudioTrack streaming.

Notes

• Requires Speech and Microphone permissions.
• Native Talk uses the active Gateway session and falls back to history polling only when response events are unavailable.
• The gateway resolves Talk playback through talk.speak using the active Talk provider. Android falls back to local system TTS only when that RPC is unavailable.
• macOS local MLX playback uses the bundled openclaw-mlx-tts helper when present, or an executable on PATH. Set OPENCLAW_MLX_TTS_BIN to point at a custom helper binary during development.
• Voice directive value ranges for ElevenLabs: stability, similarity, and style accept 0..1; speed accepts 0.5..2; latency_tier accepts 0..4.

Related

• Voice wake
• Audio and voice notes
• Media understanding