Model Online / On-Site
Duration
Modules 5 modules

Course Objective & Scope

Because cascaded voice pipelines transcribe speech into plain text first, they strip away hesitation, urgency and emotion in the caller's voice while stacking latency across three separate models. Native Speech-to-Speech (S2S) models and autonomous voice agents process and generate audio directly to overcome these limits.

In this advanced course participants build a tool-calling voice agent using OpenAI Realtime (gpt-realtime), Gemini Live native audio and hybrid cascaded patterns in Pipecat or LiveKit Agents, complete with a contact-centre test and measurement plan.

Who Should Attend?

  • AI engineers building advanced voice assistants and contact centre automation
  • Real-time audio and communications infrastructure (WebRTC/SIP) developers
  • Conversational AI architects
  • Prerequisite: Python/TypeScript, async programming and basic audio/LLM API knowledge

Key Takeaways

  • Compare native Speech-to-Speech against cascaded STT→LLM→TTS across latency, controllability and cost
  • Steer vocal tone, pacing and paralinguistic awareness in S2S models
  • Trigger deterministic business logic and tool calls reliably during live audio sessions
  • Apply hybrid patterns (native audio input with text guardrail + TTS output) where strict compliance is required
  • Deploy a multi-step voice agent with Pipecat or LiveKit Agents

Curriculum

How Native Speech-to-Speech (S2S) Models Work
  • Acoustic and semantic audio tokens and direct speech-to-speech inference
  • Paralinguistic perception (hesitation, stress, emotion) and natural vocal dynamics
Native S2S vs Cascaded vs Hybrid Architectures
  • Comparing cost, auditability, custom-noun accuracy and latency
  • Hybrid pattern: native audio understanding with text guardrails and controlled TTS
Voice Agent Orchestration and Tool Calling
  • State machines and task handoffs with Pipecat and LiveKit Agents
  • Identity verification, DTMF keypad handling and external API invocations
Multi-Agent Call Handoffs and Voice Guardrails
  • Seamless transfer from triage agent to specialist (support/billing) voice agents
  • Audio and transcript guardrails against vocal hallucination, off-brand tone and injection
Lab: Live Voice Agent and Measurement Plan
  • Working real-time voice assistant on an enterprise scenario
  • Call completion rate, vocal naturalness and cost-per-minute analysis

Other Courses in This Category

Get in touch