Model Online / On-Site
Duration
Modules 6 modules

Talk to Your Tables and Images with Multimodal RAG

In today's era of applied AI, organizations struggle to access and extract meaning from comprehensive information enriched not only with text but also with visual and tabular data. While traditional RAG architectures are text-centric, real-world documents (manuals, reports, technical drawings) contain complex visual and tabular structures. This fast-paced 3-hour "Multimodal Retrieval-Augmented Generation (RAG) with LangChain Workshop" offers participants both the theory and practice of implementing a multimodal RAG architecture from scratch.

The workshop covers the core concepts of Multimodal RAG, how to orchestrate different modalities (text, image, table) with LangChain, and aims to build a live, working chatbot capable of understanding complex documents by the end of the day. Participants will experience the end-to-end chain of multimodal data extraction, diverse embedding models, unified vector search, and prompt engineering, directly observing the power of accessing enriched information that goes beyond plain text.

Who Should Attend

Who Should Attend

  • Data scientists, MLOps engineers, software developers – those looking to build LLM-based applications capable of extracting meaning from complex, multimodal documents.

  • Technical leaders aiming to bring the practical benefits of Multimodal RAG architecture to their organization.

  • Professionals seeking tangible outputs in a short time through a learn-build-demo cycle.

Delivering intensive learning within a focused timeframe, this workshop will rapidly advance participants on the RAG journey from "multimodal documents to actionable intelligence" and ensure you leave with a live-demoed project built on real data.

Curriculum

Introduction to Multimodal RAG

Why Multimodal RAG?

  • Core RAG recap

  • Uncovering insights from documents (manuals, reports, pages with diagrams)

  • Workshop goal: Building a chatbot that understands text, images, and tables

Deconstructing the PDF: Beyond Plain Text

  • Understanding PDF structure: text streams, image objects, vector graphics, and embedded table data

  • PDF parsing tools

  • Data extraction challenges for each modality

Multimodal RAG Workflow

  • Distinct processing pipelines for text, images, and tables

  • Modality-specific embedding models vs. unified multimodal embeddings

  • Storing and retrieving multimodal information

  • Presenting combined context to the LLM
Extracting and Embedding Images and Tables

Image Extraction and Embedding

  • Extracting images from a sample PDF

  • Introduction to visual embedding models (CLIP, OpenAI models, etc.)

  • Hands-on: Generating embeddings for extracted images

  • Storing image embeddings and associated metadata (source page, image ID)

Table Extraction & Representation for RAG

  • Extracting tables from documents

  • Strategies for representing tables:

    • Embedding as structured text (Markdown, CSV string)

    • Embedding as an image for complex/graphical tables

  • Brief discussion on using an LLM to summarize tables concisely

  • Hands-on: Converting an extracted table to Markdown and generating embeddings

  • Text Extraction & Embedding

  • Brief recap of text chunking and embedding

  • Ensuring metadata consistency across modalities (such as page numbers)
Multimodal Vector Store & Retrieval Strategy

Vector Store Design

  • Option 1: A single vector store with metadata distinguishing modalities

  • Option 2: Separate vector stores for each modality

  • Pros and cons (simplicity vs. specialized search)

  • Hands-on: Setting up a Vector Database storing text, image embeddings, and table representations with metadata

Multimodal Retrieval: Finding the Right Pieces

  • Cross-modality querying:

    • Simple approach: Querying all relevant vector stores/indexes and merging results

    • LangChain's MultiVectorRetriever concept

  • Retrieving relevant text chunks, image references/descriptions, and table snippets

  • Merging and ranking results: weighting scores from different sources or simply combining top-k from each modality

  • Hands-on: Implementing a retrieval function that fetches top-k text, top-k image references, and top-k table snippets for a user query
Prompting for Multimodal RAG

Representing Multimodal Context in Prompts

  • How to "show" images to an LLM when not using a vision-capable LLM: image descriptions, captions, or placeholders (e.g., [Image: chart_on_page_5.png ...])

  • Integrating tabular data (Markdown) directly into the prompt

  • Prompt structures that guide the LLM to synthesize all context types

Effective Prompts for Multimodal Q&A

  • "Based on the text and the provided [Image: diagram_1], explain how X works."

  • "Summarize the key findings from [Table: financial_summary] and the surrounding text."

  • "What does the chart on page 7 show about user growth?" (Mapping page → image)

  • Iterative prompt refinement for desired outputs
Building the UI with Gradio

Ingesting Data into the System:

  • User uploads a PDF

  • Data Extraction (text, image, table)

  • Embedding generation

  • Storing in the vector store

Retrieval & Generation:

  • Receive user query

  • Perform multimodal retrieval

  • Construct the prompt with retrieved context

  • Call the LLM

  • Display the response

Live Demo & Testing

  • Testing the application with a sample PDF containing text, charts (images), and tables

  • Asking questions that require synthesizing information across modalities

  • Debugging
Wrap-Up

  • Summary of what was accomplished

  • Q&A

Other Courses in This Category

  • Retrieval-Augmented Generation (RAG) Fundamentals

    Hands-on fundamentals for building an end-to-end RAG system over enterprise documents: ingestion, chunking, embeddings, vector retrieval, grounded prompting with citations and baseline evaluation.

    RAG & Enterprise Knowledge Systems Corporate
    Live Course
    6 modules
    View
  • Embeddings and Semantic Search Engineering

    Deep dive into dense, sparse (SPLADE/BM25), Matryoshka and late-interaction (ColBERT) embeddings, Turkish/multilingual retrieval quality, domain adaptation and semantic search engineering.

    RAG & Enterprise Knowledge Systems Corporate
    Live Course
    6 modules
    View
  • Vector Database Architectures: pgvector, Milvus, FAISS, Weaviate and Pinecone

    Engineering deep dive into ANN indexing (HNSW, IVF, DiskANN, PQ) and vector databases (pgvector, Milvus, Weaviate, FAISS, Pinecone), covering filtered search, multi-tenancy and capacity planning.

    RAG & Enterprise Knowledge Systems Corporate
    Live Course
    6 modules
    View
  • Data Preparation, Document Parsing and Chunking for RAG

    Parsing complex PDFs, tables and scanned enterprise documents with Docling, Unstructured and VLMs, and applying structural, semantic, parent-child and contextual chunking for high-recall RAG.

    RAG & Enterprise Knowledge Systems Corporate
    Live Course
    6 modules
    View
  • Citations, Grounding and Hallucination Control in RAG Systems

    Engineering RAG systems for strict grounding, verifiable inline citations, conflict resolution across sources, calibrated abstention when evidence is missing and automated faithfulness verification.

    RAG & Enterprise Knowledge Systems Corporate
    Live Course
    6 modules
    View
  • Advanced RAG Advanced RAG

    Enterprise RAG architectures covering BM25-dense hybrid retrieval, ColBERT, neural reranking, self-correcting Agentic RAG, GraphRAG, Text-to-SQL, and GPU-accelerated serving.

    RAG & Enterprise Knowledge Systems Corporate, Individual
    Live Course
    10 modules
    View

Get in touch