Course Objective & Scope
Enterprise automation often requires understanding both dense technical documents and complex desktop or web application screens, yet general-purpose VLMs frequently miss small icons, tight data grids and exact click coordinates on their own. This advanced programme treats document and screen understanding as a unified visual-structural parsing problem.
Participants use OmniParser, Qwen-VL, Florence-2 and Docling to build pipelines for UI widget detection, icon captioning, Set-of-Marks (SoM) grounding and dense table/diagram extraction, delivering a working prototype on their organisation's screens and documents.
Who Should Attend?
- AI engineers building computer-use, RPA and document intelligence engines
- Computer vision and multimodal model developers
- R&D and enterprise automation platform teams
- Prerequisite: Python, image processing basics and VLM/LLM API experience
Key Takeaways
- Detect interactive UI elements (buttons, icons, inputs, grid cells) on application screens with high precision
- Translate pixel coordinates into semantic element maps using OmniParser and Set-of-Marks
- Visually parse remote desktop (Citrix/RDP) and legacy screens where no DOM or accessibility tree exists
- Extract structured schemas from embedded PDFs, charts and dense data tables displayed on screen
- Benchmark pointing accuracy, widget recognition recall and parsing latency
Curriculum
Anatomy of Visual-Textual Interfaces: Documents and Screens
- How dense screens and documents differ from natural photography
- Challenges of high resolution, tiny fonts, cryptic icons and hierarchical layouts
Screen Parsing and OmniParser Pipelines
- Detecting interactable regions and captioning icon functionality
- Generating Set-of-Marks (SoM) overlays and structured screen state representations
Visual Grounding and Spatial Models
- Pinpointing click targets with Qwen-VL, UI-TARS and dedicated grounding models
- Handling multi-monitor setups, scrollbars and transient dropdown states
Dense Document, Grid and Technical Diagram Parsing
- Reading PDFs, invoices and engineering schematics rendered inside applications
- Cross-checking merged cells, form fields and chart values
End-to-End Lab and Accuracy Benchmarking
- Unified perception prototype on enterprise ERP and document screens
- Internal ScreenSpot/DocVQA-style evaluation and latency measurement plan
Other Courses in This Category
-
Evaluation engineering for multimodal systems across vision, documents, audio and video: OCR/extraction metrics, grounding IoU, speech WER/MOS and turn-taking metrics, VLM-as-a-judge and regression pipelines.
- Live Course
- 5 modules
-
Comprehensive introduction to multimodal AI models that understand and generate text, images, audio and video: model families, enterprise use cases, capability limits, cost/latency trade-offs and model selection.
- Live Course
- 5 modules
-
End-to-end design of enterprise applications combining text, image, audio and document inputs: media preprocessing, structured outputs, hybrid model routing and cost/latency optimisation.
- Live Course
- 5 modules
-
High-accuracy extraction from complex PDFs, invoices, contracts, tables and forms using VLMs and modern document parsers: layout understanding, schema validation, bounding-box grounding and human-in-the-loop review.
- Live Course
- 5 modules
-
Multimodal RAG architecture over enterprise documents containing tables, charts, diagrams and images: visual-text indexing, ColPali and image embeddings, hybrid retrieval and visually grounded answer generation.
- Live Course
- 5 modules
-
Conversation design for LLM-based chat and voice assistants: persona and tone guidelines, multi-turn dialogue flows, disambiguation, repair strategies and seamless handoff to human agents.
- Live Course
- 5 modules
Get in touch
