This guide demonstrates how to assemble an AI voice agent device utilizing realtime AI speech capabilities from OpenAI's Realtime API, combined with an ESP32 microcontroller, secure WebSockets, and Deno Edge Functions to support uninterrupted global dialogues exceeding 10 minutes.

A live version of this README can be found at ElatoAI.

⚡️ DIY Hardware Design

The reference design employs an ESP32-S3 microcontroller paired with a few essential components:

Required Components:

• ESP32-S3 development board
• I2S microphone (e.g., INMP441)
• I2S amplifier and speaker (e.g., MAX98357A)
• Push button to initiate or end conversations
• RGB LED for status indication
• Optional: touch sensor for hands-free control

Hardware options: A pre-assembled PCB and complete device are offered in the ElatoAI store.

📱 App Design

Manage your ESP32 AI device via a mobile-friendly webapp from your phone.

• Choose from various AI characters
• Engage in realtime dialogues with instant AI replies
• Design bespoke AI characters

✨ Quick Start Tutorial

• Clone the repository

Visit the ElatoAI GitHub repository and clone it.

git clone https://github.com/akdeb/ElatoAI.git
cd ElatoAI

• Set your environment variables (OPENAI_API_KEY, SUPABASE_ANON_KEY)

In the frontend-nextjs directory, create a .env.local file and configure your environment variables.

cd frontend-nextjs
cp .env.example .env.local
 
# In .env.local, set your environment variables 
# NEXT_PUBLIC_SUPABASE_ANON_KEY=<your-supabase-anon-key>
# OPENAI_API_KEY=<your-openai-api-key>

In the server-deno directory, create a .env file and configure your environment variables.

cd server-deno
cp .env.example .env
 
# In .env, set your environment variables 
# SUPABASE_KEY=<your-supabase-anon-key>
# OPENAI_API_KEY=<your-openai-api-key>

• Start Supabase

Install the Supabase CLI and initialize your local Supabase backend. From the project root, execute:

brew install supabase/tap/supabase
supabase start # Starts your local Supabase server with the default migrations and seed data.

• Set up your NextJS Frontend

(See the Frontend README)

From the frontend-nextjs directory, execute these commands. (Login creds: Email: admin@elatoai.com, Password: admin)

cd frontend-nextjs
npm install
 
# Run the development server
npm run dev

• Start the Deno server

(See the Deno server README)

# Navigate to the server directory
cd server-deno
 
# Run the server at port 8000
denno run -A --env-file=.env main.ts

• Setup the ESP32 Device firmware

(See the ESP32 Device README)

In Config.cpp set ws_server and backend_server to your local IP address. Run ifconfig in your console and locate en0 -> inet -> 192.168.1.100 (it may vary based on your WiFi network). This directs the ESP32 to connect to your local NextJS frontend and Deno server. Ensure all services are on the same WiFi network.

• Setup the ESP32 Device Wifi

Build and upload the firmware to your ESP32 device. The ESP32 will launch an ELATO-DEVICE captive portal for WiFi setup. Connect to it and navigate to http://192.168.4.1 to input your WiFi credentials.

After configuring WiFi, power cycle the device (OFF then ON), and it will join your WiFi and connect to the server.

You're now ready to converse with your AI Character!

🚀 Ready to Launch?

• Register your device by entering your ESP32 Device's MAC Address and a unique user code into the devices table in Supabase.

Pro Tip: To retrieve your ESP32-S3 Device's MAC Address, build and upload test/print_mac_address_test.cpp using PlatformIO and check the serial monitor.

In the frontend client's Settings page, input the unique user code to associate the device with your Supabase account.

For local testing, enable the DEV_MODE macro in firmware-arduino/Config.h and the Deno server environment variable to utilize local IP addresses.

Repeat the process to register additional devices to your account.

Project Architecture

ElatoAI comprises three primary components:

• Frontend Client (Next.js hosted on Vercel) - for creating and interacting with AI agents, then directing them to your ESP32 device

• Edge Server Functions (Deno on Deno/Supabase Edge) - managing WebSocket connections from the ESP32 and OpenAI API interactions

• ESP32 IoT Client (PlatformIO/Arduino) - handling WebSocket links from the Edge Server and relaying audio to the OpenAI API through the Deno edge server.

🌟 Key Features

• Realtime Speech-to-Speech: Lightning-fast speech conversion via OpenAI's Realtime APIs.

• Create Custom AI Agents: Design agents with unique personalities and voices.

• Customizable Voices: Select from diverse voices and character traits.

• Secure WebSockets: Dependable, encrypted WebSocket links.

• Server VAD Turn Detection: Smart handling of dialogue flow for fluid exchanges.

• Opus Audio Compression: Superior audio streaming at low bandwidth.

• Global Edge Performance: Deno Edge Functions deliver low-latency worldwide.

• ESP32 Arduino Framework: Streamlined hardware integration.

• Conversation History: Access past conversation logs.

• Device Management and Authentication: Register and oversee your devices.

• User Authentication: Robust authentication and access control.

• Conversations with WebRTC and Websockets: Use WebRTC on the NextJS webapp or WebSockets on ESP32 for AI chats.

• Volume Control: Adjust ESP32 speaker volume remotely via NextJS webapp.

• Realtime Transcripts: Conversation transcripts saved in Supabase DB in realtime.

• OTA Updates: Firmware updates over the air for ESP32.

• Wifi Management with captive portal: WiFi configuration directly from the ESP32.

• Factory Reset: Reset ESP32 to factory settings from NextJS webapp.

• Button and Touch Support: Control via button or touch sensor.

• No PSRAM Required: Runs speech-to-speech AI without PSRAM on ESP32.

• OAuth for Web client: OAuth integration for user management of AI characters and devices.

🛠 Tech Stack

📈 Core Use Cases

Refer to Usecases.md for detailed core use cases applicable to the Elato AI device or similar custom conversational AI hardware.

🗺️ High-Level Flow

[Image placeholder]

Project Structure

[Image placeholder]

⚙️ PlatformIO Config

[env:esp32-s3-devkitc-1]
platform = espressif32 @ 6.10.0
board = esp32-s3-devkitc-1
framework = arduino
monitor_speed = 115200
 
lib_deps =
    bblanchon/ArduinoJson@^7.1.0
    links2004/WebSockets@^2.4.1
    ESP32Async/ESPAsyncWebServer@^3.7.6
    https://github.com/esp-arduino-libs/ESP32_Button.git#v0.0.1
    https://github.com/pschatzmann/arduino-audio-tools.git#v1.0.1
    https://github.com/pschatzmann/arduino-libopus.git#a1.1.0

📊 Important Stats

• ⚡️ Latency: <2s round-trip globally

• 🎧 Audio Quality: Opus codec at bitrate 12kbps (high clarity)

• ⏳ Uninterrupted Conversations: Up to 10 minutes continuous conversations

• 🌎 Global Availability: Optimized with edge computing with Deno

🛡 Security

• Secure WebSockets (WSS) for encrypted data transfers

• Optional: API Key encryption with 256-bit AES

• Supabase DB for secure authentication

• Supabase RLS for all tables

🚫 Limitations

• 3-4s Cold start time while connecting to edge server

• Limited to upto 10 minutes of uninterrupted conversations

• Edge server stops when wall clock time is exceeded

• No speech interruption detection on ESP32

License

This project is licensed under the MIT License - see the LICENSE file for details.

This example is part of the OpenAI Cookbook. For the full project and latest updates, check out ElatoAI and consider giving it a ⭐️ if you find it useful!