Course Objective & Scope
Standard text-only RAG pipelines lose critical information when they flatten PDF charts, architecture diagrams, product visuals and complex tables into plain text. This course covers Multimodal RAG architectures that index and retrieve visual and textual content together.
Participants implement two core paradigms: indexing VLM-generated rich summaries linked to raw image/table assets (multi-vector indexing), and retrieving directly over page images with ColPali and multimodal embeddings. They leave with a working prototype on enterprise documents and a quality measurement plan.
Who Should Attend?
- AI engineers extending existing RAG systems to handle tables, charts and visuals
- Data and software teams building search over technical manuals, financial reports and catalogues
- Enterprise knowledge management architects
- Prerequisite: Python and experience with text-based RAG and vector databases
Key Takeaways
- Parse and chunk text, tables and figures without losing structural context
- Build multi-vector indexes pairing VLM visual/table summaries with raw assets
- Implement OCR-free visual page retrieval using ColPali and multimodal embeddings
- Feed retrieved page and diagram images into a VLM at query time for grounded answers
- Create an evaluation dataset measuring multimodal retrieval and answer faithfulness
Curriculum
Why Text-Only RAG Fails on Visual Documents
- Information loss on tables, charts, flowcharts and technical drawings
- Comparing Multimodal RAG architectural patterns
Multimodal Parsing and Multi-Vector Indexing
- Separating text, tables (HTML/Markdown) and figure regions
- Generating VLM captions for visuals/tables while storing raw assets
Visual Embeddings and the ColPali Paradigm
- CLIP/SigLIP and late-interaction ColPali/ColQwen models
- Indexing complex PDF pages directly as images without fragile OCR
Hybrid Retrieval, Reranking and Grounded Generation
- Fusing text and visual hits and applying multimodal reranking
- Passing retrieved page crops into a VLM for cited answers
Multimodal RAG Evaluation and Optimisation
- Curating benchmark sets with chart, table and diagram queries
- Balancing retrieval recall, answer faithfulness, storage and latency
Other Courses in This Category
-
Conversation design for LLM-based chat and voice assistants: persona and tone guidelines, multi-turn dialogue flows, disambiguation, repair strategies and seamless handoff to human agents.
- Live Course
- 5 modules
-
Designing AI customer service agents and human-agent assist systems powered by knowledge bases (RAG), CRM/order tools and policy guardrails, from contact centre architecture to production rollout.
- Live Course
- 5 modules
-
Speech-to-Text (STT), LLM and Text-to-Speech (TTS) pipelines for enterprise voice AI: low-latency audio processing with Whisper, Deepgram, ElevenLabs and Cartesia, acoustic challenges and telephony integration.
- Live Course
- 5 modules
-
Real-time AI assistants that interact instantaneously over voice, camera and screen sharing: full-duplex streaming, barge-in handling, live visual context, in-flight tool calling and resilient session management.
- Live Course
- 5 modules
-
Computer-use systems that operate desktop and browser environments via screenshot perception, coordinate grounding and keyboard/mouse actions: Claude Computer Use, OpenAI CUA, sandboxed VMs and safe execution.
- Live Course
- 5 modules
-
UI agents that execute workflows across web portals and legacy enterprise screens without APIs: DOM plus visual grounding (Set-of-Marks) with Playwright, Stagehand and Browserbase, session handling and resilient automation.
- Live Course
- 5 modules
Get in touch
