Course Objective & Scope
Because cascaded voice pipelines transcribe speech into plain text first, they strip away hesitation, urgency and emotion in the caller's voice while stacking latency across three separate models. Native Speech-to-Speech (S2S) models and autonomous voice agents process and generate audio directly to overcome these limits.
In this advanced course participants build a tool-calling voice agent using OpenAI Realtime (gpt-realtime), Gemini Live native audio and hybrid cascaded patterns in Pipecat or LiveKit Agents, complete with a contact-centre test and measurement plan.
Who Should Attend?
- AI engineers building advanced voice assistants and contact centre automation
- Real-time audio and communications infrastructure (WebRTC/SIP) developers
- Conversational AI architects
- Prerequisite: Python/TypeScript, async programming and basic audio/LLM API knowledge
Key Takeaways
- Compare native Speech-to-Speech against cascaded STT→LLM→TTS across latency, controllability and cost
- Steer vocal tone, pacing and paralinguistic awareness in S2S models
- Trigger deterministic business logic and tool calls reliably during live audio sessions
- Apply hybrid patterns (native audio input with text guardrail + TTS output) where strict compliance is required
- Deploy a multi-step voice agent with Pipecat or LiveKit Agents
Curriculum
How Native Speech-to-Speech (S2S) Models Work
- Acoustic and semantic audio tokens and direct speech-to-speech inference
- Paralinguistic perception (hesitation, stress, emotion) and natural vocal dynamics
Native S2S vs Cascaded vs Hybrid Architectures
- Comparing cost, auditability, custom-noun accuracy and latency
- Hybrid pattern: native audio understanding with text guardrails and controlled TTS
Voice Agent Orchestration and Tool Calling
- State machines and task handoffs with Pipecat and LiveKit Agents
- Identity verification, DTMF keypad handling and external API invocations
Multi-Agent Call Handoffs and Voice Guardrails
- Seamless transfer from triage agent to specialist (support/billing) voice agents
- Audio and transcript guardrails against vocal hallucination, off-brand tone and injection
Lab: Live Voice Agent and Measurement Plan
- Working real-time voice assistant on an enterprise scenario
- Call completion rate, vocal naturalness and cost-per-minute analysis
Other Courses in This Category
-
Streaming infrastructure engineering for real-time AI applications: WebRTC, WebSockets and SSE protocols, OpenAI Realtime and Gemini Live API integration, audio/video buffer management and horizontal scaling.
- Live Course
- 5 modules
-
Human-like turn-taking and millisecond latency engineering in voice AI: acoustic VAD, semantic endpointing, false-interruption suppression, TTFT/TTFB optimisation and speculative execution.
- Live Course
- 5 modules
-
Parsing dense documents and application screens (GUIs) at the pixel, coordinate and structural-tree level: OmniParser, Qwen-VL, Docling, UI widget detection and unified document/screen understanding models.
- Live Course
- 5 modules
-
Evaluation engineering for multimodal systems across vision, documents, audio and video: OCR/extraction metrics, grounding IoU, speech WER/MOS and turn-taking metrics, VLM-as-a-judge and regression pipelines.
- Live Course
- 5 modules
-
Comprehensive introduction to multimodal AI models that understand and generate text, images, audio and video: model families, enterprise use cases, capability limits, cost/latency trade-offs and model selection.
- Live Course
- 5 modules
-
End-to-end design of enterprise applications combining text, image, audio and document inputs: media preprocessing, structured outputs, hybrid model routing and cost/latency optimisation.
- Live Course
- 5 modules
Get in touch
