Introduction

Voice-enabled AI agents are transforming user interactions, enabling hands-free conversations for applications like virtual assistants, customer support bots, and interactive demos. This tutorial shows you how to build a real-time voice AI agent using the Claude API (via Anthropic's Python SDK) combined with the browser's Web Speech API for speech recognition and synthesis.

We'll create a web application where users speak, the browser transcribes in real-time, Claude generates contextual responses (maintaining conversation history), and the agent speaks back instantly via streaming. This leverages Claude 3.5 Sonnet's superior reasoning for natural, coherent dialogues.

Key features:

• Continuous listening with interim transcription results
• WebSocket for low-latency communication
• Streaming Claude responses for real-time TTS
• Conversation history for agent-like persistence
• Pure browser APIs—no external STT/TTS services needed

Perfect for developers building Claude-powered voice apps in marketing demos, HR chatbots, or sales assistants.

Prerequisites

Before starting:

• Python 3.10+
• Anthropic API key (free tier available)
• Basic knowledge of Python, HTML, and JavaScript
• A modern browser (Chrome/Edge for best Web Speech support)

Install dependencies:

pip install fastapi uvicorn anthropic

Project Structure

Create a new directory:

mkdir claude-voice-agent
cd claude-voice-agent

You'll need two files:

• main.py: FastAPI backend with WebSocket endpoint
• index.html: Frontend with speech handling

Backend: FastAPI with Claude Integration

FastAPI handles WebSockets for real-time bidirectional communication. We'll maintain conversation history per connection using an in-memory dict (scale to Redis for production).

Create main.py:

import os
import uuid
from typing import Dict, List, Any
from fastapi import FastAPI, WebSocket, WebSocketDisconnect
from fastapi.staticfiles import StaticFiles
from fastapi.responses import HTMLResponse
from anthropic import Anthropic, HUMAN_PROMPT, AI_PROMPT

app = FastAPI()
app.mount "/static", StaticFiles(directory="."), name="static"

client = Anthropic(api_key=os.getenv("ANTHROPIC_API_KEY"))

# In-memory sessions: {connection_id: history}
sessions: Dict[str, List[Dict[str, str]]] = {}

@app.get("/")
async def get_index():
    with open("index.html") as f:
        return HTMLResponse(f.read())

@app.websocket("/ws")
async def websocket_endpoint(websocket: WebSocket):
    await websocket.accept()
    session_id = str(uuid.uuid4())
    sessions[session_id] = []

    try:
        while True:
            data = await websocket.receive_text()
            user_message = data.strip()

            # Append user message to history
            history = sessions[session_id]
            history.append({"role": "user", "content": user_message})

            # Call Claude with streaming
            stream = client.messages.stream(
                model="claude-3-5-sonnet-20240620",
                max_tokens=1024,
                messages=history,
                temperature=0.7,
            )

            full_response = ""
            async for text in stream.text_stream():
                full_response += text
                await websocket.send_text(text)  # Stream tokens

            # Append AI response to history
            history.append({"role": "assistant", "content": full_response})

    except WebSocketDisconnect:
        del sessions[session_id]

if __name__ == "__main__":
    import uvicorn
    uvicorn.run(app, host="127.0.0.1", port=8000)

Key notes:

• Set ANTHROPIC_API_KEY env var: export ANTHROPIC_API_KEY=your_key_here
• Uses claude-3-5-sonnet-20240620 for best performance (swap for Opus/Haiku as needed)
• Streaming via messages.stream() and text_stream() for real-time tokens
• History format matches Anthropic's messages API—no need for legacy HUMAN_PROMPT

Frontend: Web Speech API Integration

The browser handles STT (SpeechRecognition) and TTS (SpeechSynthesis). We connect via WebSocket for sending transcripts and receiving streamed responses.

Create index.html:

<!DOCTYPE html>
<html>
<head>
    <title>Claude Voice Agent</title>
    <style>
        body { font-family: Arial; max-width: 800px; margin: 0 auto; padding: 20px; }
        #chat { border: 1px solid #ccc; height: 400px; overflow-y: scroll; padding: 10px; }
        button { padding: 10px 20px; font-size: 16px; }
        .user { color: blue; }
        .ai { color: green; }
    </style>
</head>
<body>
    <h1>Claude Voice AI Agent</h1>
    <div id="chat"></div>
    <button id="start">Start Listening</button>
    <button id="stop">Stop</button>

    
</body>
</html>

Speech API details:

• interimResults=true for real-time transcription previews
• Only send final transcripts to Claude to avoid noise
• Streaming TTS: Cancel and respeak full response on each token for fluid playback
• Mic permission auto-prompted on first start()

Running the App

1. Set API key: export ANTHROPIC_API_KEY=sk-ant-...
2. Run server: uvicorn main:app --reload
3. Open http://127.0.0.1:8000
4. Click Start Listening, grant mic access, and speak!

Example interaction:

• User: "What's the weather like?"
• Claude: Streams response like "I don't have real-time data, but..."
• Spoken back instantly

Enhancements for Production

• Context Management: Limit history to last 10 exchanges (history = history[-10:])
• System Prompt: Add initial message for agent persona:

  messages = [{"role": "system", "content": "You are a helpful voice assistant."}] + history
  

• Error Handling: Retry on API failures, validate transcripts
• Streaming Input: For ultra-real-time, send interim to Claude via tool calls (advanced MCP integration)
• Deploy: Vercel/Render for FastAPI; add CORS if needed
• Multi-Turn Polish: Clear interim previews after final
• Custom Voices: Use SpeechSynthesisVoice selection
• Mobile: Add webkitSpeechRecognition fallbacks

Compare to GPT: Claude excels in coherent multi-turn convos without hallucinating history.

Troubleshooting

Conclusion

You've built a fully functional voice AI agent with Claude API—real-time, conversational, and extensible. This foundation scales to enterprise playbooks like sales demos (integrate Zapier) or engineering tools (Claude Code voice commands).

Fork on GitHub, experiment with Haiku for speed, or Opus for depth. Share your builds in Claude Directory comments!

Word count: ~1450