Building Enterprise Realtime Voice Agents from Scratch (2026)

Free

Salesforce AI Research: complete tutorial for production voice agents — cascaded streaming pipeline (STT→LLM→TTS), ~750ms TTFA, function calling, full open-source codebase with 9 chapters

FreeFree tier

Inputs: audioOutputs: audio

Type

Open Source

Visit Website GitHub (40 stars)

About Building Enterprise Realtime Voice Agents from Scratch (2026)

This technical tutorial from Salesforce AI Research provides a comprehensive, step-by-step guide to building enterprise-grade realtime voice agents using a cascaded streaming pipeline (Speech-to-Text → LLM → Text-to-Speech). The tutorial evaluates why end-to-end speech-to-speech models are not yet practical for self-hosted deployments, using Qwen3-Omni as a case study (cloud API achieves ~702ms but not self-hostable; local vLLM lacks audio synthesis; local Transformers full pipeline is ~146s). The recommended architecture uses Deepgram for streaming STT, vLLM-served LLMs with function calling for streaming text generation, and ElevenLabs for streaming TTS, achieving a measured time-to-first-audio (TTFA) of 755ms (best case 729ms). The full open-source codebase is released as a 9-chapter progressive tutorial with working, tested code for every component.

Key Features

Cascaded streaming pipeline (STT→LLM→TTS) for self-hosted voice agents

Measured ~755ms time-to-first-audio (best case 729ms) with function calling

Uses Deepgram streaming STT, vLLM-served LLM, and ElevenLabs streaming TTS

Full function calling support for enterprise integrations

9-chapter progressive tutorial with working, tested open-source code

Evaluates alternatives: Qwen3-Omni cloud API (~702ms), local vLLM, and local Transformers

Designed for enterprise-grade production deployment, not just prototypes

Pros & Cons

Pros

Practical step-by-step tutorial with fully working open-source code
Achieves low latency (~755ms TTFA) suitable for realtime use
Modular architecture allows swapping components (STT/LLM/TTS)
Includes function calling, a key requirement for enterprise actions
Transparent evaluation of end-to-end alternatives with concrete numbers

Cons

Not a true end-to-end speech-to-speech model – relies on cascaded components
Requires external proprietary services (Deepgram, ElevenLabs) for STT and TTS
Latency of 755ms may still be noticeable in highly interactive scenarios
Tutorial focuses on specific tools; may not cover all production concerns (scaling, error handling)
Self-hosted LLM component (vLLM) requires significant compute resources

Best For

Building enterprise-grade realtime voice agents from scratchDeploying self-hosted voice AI systems with low latencyDeveloping voice assistants with function calling capabilitiesEducational resource for understanding cascaded speech architecturesPrototyping and productionizing voice-based customer service bots

FAQ

What latency does the voice agent achieve?

The tutorial reports a measured time-to-first-audio (TTFA) of 755ms, with a best case of 729ms, using the cascaded pipeline with Deepgram, vLLM, and ElevenLabs.

What components are used in the tutorial?

The tutorial uses Deepgram for streaming speech-to-text, vLLM-served LLMs with function calling for text generation, and ElevenLabs for streaming text-to-speech.

Is the code open-source?

Yes, the full codebase is released as an open-source 9-chapter progressive tutorial with working, tested code for every component.

Why not use an end-to-end speech-to-speech model?

The tutorial explains that while end-to-end models like Qwen3-Omni show promise, fully self-hosted end-to-end solutions are not yet available. Their cloud API is not self-hostable, local vLLM lacks audio synthesis, and local Transformers full pipeline is far too slow (~146s) for realtime use.

Who authored this tutorial?

The tutorial is authored by Jielin Qiu, Zixiang Chen, Liangwei Yang, and others from Salesforce AI Research, as indicated on the arXiv paper.