Course Objective & Scope
Real-time assistants—where a user can interrupt mid-sentence or point a camera at faulty equipment and get immediate spoken guidance—cannot be built with classic request–response patterns. This course covers the architecture of always-on, full-duplex voice and vision assistants.
Using the OpenAI Realtime API, Gemini Multimodal Live and LiveKit Agents, participants implement barge-in handling, non-blocking background tool calling and live camera/screen frame ingestion, producing a real-time assistant prototype and measurement plan tailored to an organisational scenario.
Who Should Attend?
- AI and full-stack engineers building live voice and vision assistants
- Teams developing field service support, remote expert assistance, coaching or customer interaction products
- Real-time communications architects
- Prerequisite: Python or TypeScript and asynchronous/event-driven programming experience
Key Takeaways
- Build always-on, full-duplex real-time assistant sessions
- Flush audio buffers and reconcile conversation state cleanly when the user barges in
- Synchronise live camera or screen-share frames with ongoing spoken dialogue
- Execute asynchronous tool calls in the background without freezing the conversation
- Design session recovery, latency monitoring and cost controls for live sessions
Curriculum
From Request–Response to Full-Duplex Live Sessions
- Stateful live session architecture and event-driven message loops
- OpenAI Realtime, Gemini Live and open-source orchestration layers
Natural Dialogue Dynamics and Barge-In
- Halting audio playback and clearing client buffers instantly on interruption
- Truncating interrupted assistant turns accurately in the conversation state
Ingesting Live Camera and Screen Context
- Frame sampling and bandwidth management from camera and screen shares
- Resolving deictic references such as 'the error on my screen right now'
In-Flight Tool Calling and Background Tasks
- Keeping the conversation natural with holding cues during slow API calls
- Injecting asynchronous tool results back into an active voice session
Resilience, Cost and Quality Measurement
- Reconnection, session handoff and synchronized audio/transcript logging
- Real-time assistant prototype and end-to-end latency measurement plan
Other Courses in This Category
-
Computer-use systems that operate desktop and browser environments via screenshot perception, coordinate grounding and keyboard/mouse actions: Claude Computer Use, OpenAI CUA, sandboxed VMs and safe execution.
- Live Course
- 5 modules
-
UI agents that execute workflows across web portals and legacy enterprise screens without APIs: DOM plus visual grounding (Set-of-Marks) with Playwright, Stagehand and Browserbase, session handling and resilient automation.
- Live Course
- 5 modules
-
User control and confirmation UX in voice, multimodal and UI-acting agents: action previews, tiered permissions, live takeover/handback, undo mechanisms and UI safety patterns.
- Live Course
- 5 modules
-
Testing voice, multimodal and UI-acting agents for user experience, trust and safety before launch: user trust calibration, adversarial scenarios, frustration signal detection and release criteria.
- Live Course
- 5 modules
-
Multimodal model architectures that unify vision, audio and text representations: ViT/SigLIP and audio encoders, cross-modal projection/fusion layers, and LoRA fine-tuning of open-weight VLMs (Qwen-VL, Llama, Pixtral).
- Live Course
- 5 modules
-
Building visual inspection and analysis applications with Vision-Language Models (VLMs) in manufacturing, retail, insurance and field operations: visual grounding, multi-image comparison and hybrid CV+VLM pipelines.
- Live Course
- 5 modules
Get in touch
