Course Objective & Scope
Many enterprise workflows remain locked inside supplier portals, government websites or legacy ERP/CRM screens that lack APIs, while traditional RPA scripts break on the smallest UI tweak. This course covers UI-using agents that combine the DOM, accessibility tree and screenshots to navigate web and application interfaces reliably.
Participants use Playwright, Stagehand and Set-of-Marks (SoM) visual grounding to build an agent that fills forms, navigates tables and handles file uploads/downloads, producing a human-confirmed prototype and a resilience and latency measurement plan.
Who Should Attend?
- Software and RPA engineers building web and enterprise screen automation
- AI developers implementing agentic process automation
- Operational excellence and digital transformation specialists
- Prerequisite: basic Python or TypeScript and familiarity with HTML/DOM
Key Takeaways
- Build hybrid UI agents that fuse DOM/accessibility trees with visual screen perception
- Implement natural-language selectors and self-healing steps that survive UI changes
- Manage authentication, persistent sessions and multi-step form workflows safely
- Present live previews and human confirmation gates before final submission or payment
- Optimise step completion rate, page-wait latency and token cost
Curriculum
From Brittle Selectors to Cognitive UI Agents
- Limits of XPath/CSS selectors and LLM-driven interface interaction
- Combining DOM, Accessibility Trees and Set-of-Marks (SoM) visual overlays
Browser Orchestration: Playwright, Stagehand and Browserbase
- Blending deterministic Playwright code with agentic actions
- Cloud browser sessions, proxies and parallel execution
Session Handling, Dynamic Pages and Data Extraction
- SSO/MFA handoff, state persistence and SPA wait strategies
- Extracting structured records across paginated tables and dynamic dropdowns
Self-Healing Workflows and Action Caching
- Caching discovered action paths for deterministic replay
- Invoking the VLM/LLM only when the UI changes to save cost and latency
Human Oversight and Safe Execution
- Pausing before commit buttons, visual proof logging and live takeover
- Enterprise case prototype and quality/latency measurement plan
Other Courses in This Category
-
User control and confirmation UX in voice, multimodal and UI-acting agents: action previews, tiered permissions, live takeover/handback, undo mechanisms and UI safety patterns.
- Live Course
- 5 modules
-
Testing voice, multimodal and UI-acting agents for user experience, trust and safety before launch: user trust calibration, adversarial scenarios, frustration signal detection and release criteria.
- Live Course
- 5 modules
-
Multimodal model architectures that unify vision, audio and text representations: ViT/SigLIP and audio encoders, cross-modal projection/fusion layers, and LoRA fine-tuning of open-weight VLMs (Qwen-VL, Llama, Pixtral).
- Live Course
- 5 modules
-
Building visual inspection and analysis applications with Vision-Language Models (VLMs) in manufacturing, retail, insurance and field operations: visual grounding, multi-image comparison and hybrid CV+VLM pipelines.
- Live Course
- 5 modules
-
End-to-end Speech-to-Speech (S2S) models and autonomous voice agents: acoustic emotion and prosody perception, voice tool calling, multi-agent call handoffs and production architecture with Pipecat and LiveKit.
- Live Course
- 5 modules
-
Streaming infrastructure engineering for real-time AI applications: WebRTC, WebSockets and SSE protocols, OpenAI Realtime and Gemini Live API integration, audio/video buffer management and horizontal scaling.
- Live Course
- 5 modules
Get in touch
