Course Objective & Scope
Not every enterprise system exposes a modern API; many business processes still run inside desktop or web applications accessible only through a graphical user interface (GUI). Computer-use systems enable agents to perceive screenshots visually and emit cursor movements, clicks and keystrokes to operate software like a human user.
In this course participants dissect the perception–decision–action loop of Anthropic Claude Computer Use, the OpenAI Computer-Using Agent (CUA) and open-weight GUI models, building a sandboxed, human-approved computer-use prototype and a latency/quality measurement plan.
Who Should Attend?
- AI and software engineers building next-generation automation and agent infrastructure
- Automation architects looking to move beyond brittle RPA selectors
- Platform and security engineers
- Prerequisite: Python, Docker/container basics and LLM API experience
Key Takeaways
- Decide when to use API tool calling, DOM automation or pixel-based computer use
- Set up a computer-use execution loop inside an isolated virtual display (X11/VNC/Docker or cloud desktop)
- Configure screen resolution, coordinate scaling and post-action visual verification
- Implement recovery strategies for stalls, misclicks and unexpected modal pop-ups
- Deliver a safe computer-use prototype with human confirmation points and screen-recorded audit trails
Curriculum
The Computer-Use Paradigm and Core Architecture
- Moving from brittle RPA and DOM selectors to visual screen perception
- Comparing Claude Computer Use, OpenAI CUA and OmniParser/UI-TARS approaches
Virtual Desktop and Sandbox Environment Setup
- Isolated display inside a container/VM, VNC streaming and action executors (xdotool/Playwright)
- Screen resolution standardisation and coordinate scaling
The Perceive–Act–Verify Loop
- Click, drag, keyboard shortcut and scroll actions
- Verifying screen state changes after each action and handling dynamic waits
Error Recovery and Hybrid (GUI + API) Execution
- Handling unexpected modals, slow-loading screens and loop escapes
- Using APIs/CLI where available and falling back to GUI only where necessary
Safety, Human Confirmation and Performance Measurement
- Visual prompt injection risks from untrusted on-screen content
- Human confirmation before critical clicks, per-step latency and task success metrics
Other Courses in This Category
-
UI agents that execute workflows across web portals and legacy enterprise screens without APIs: DOM plus visual grounding (Set-of-Marks) with Playwright, Stagehand and Browserbase, session handling and resilient automation.
- Live Course
- 5 modules
-
User control and confirmation UX in voice, multimodal and UI-acting agents: action previews, tiered permissions, live takeover/handback, undo mechanisms and UI safety patterns.
- Live Course
- 5 modules
-
Testing voice, multimodal and UI-acting agents for user experience, trust and safety before launch: user trust calibration, adversarial scenarios, frustration signal detection and release criteria.
- Live Course
- 5 modules
-
Multimodal model architectures that unify vision, audio and text representations: ViT/SigLIP and audio encoders, cross-modal projection/fusion layers, and LoRA fine-tuning of open-weight VLMs (Qwen-VL, Llama, Pixtral).
- Live Course
- 5 modules
-
Building visual inspection and analysis applications with Vision-Language Models (VLMs) in manufacturing, retail, insurance and field operations: visual grounding, multi-image comparison and hybrid CV+VLM pipelines.
- Live Course
- 5 modules
-
End-to-end Speech-to-Speech (S2S) models and autonomous voice agents: acoustic emotion and prosody perception, voice tool calling, multi-agent call handoffs and production architecture with Pipecat and LiveKit.
- Live Course
- 5 modules
Get in touch
