Model Online / On-Site
Duration
Modules 5 modules

Course Objective & Scope

RAG systems that rely solely on vector similarity routinely confuse last year's policy with the current one, mix up documents across product lines and retrieve chunks the user is not authorised to see. This training teaches how to engineer the metadata layer that brings precision, temporal awareness and access control to RAG and agent architectures.

Document- and chunk-level metadata schema design, automated metadata extraction with LLMs and NER models, self-querying retrievers that translate natural-language questions into structured filters, temporal validity (valid_from/valid_to), citations and ACL pre-filtering are covered hands-on. Participants design a metadata schema and filtering pipeline for their own enterprise knowledge base.

Who Should Attend?

  • AI engineers building enterprise RAG, search and knowledge assistants
  • Data engineers building document ingestion pipelines and vector indexes
  • Platform developers defining agent tools and enterprise knowledge sources
  • Prerequisite: Python, basic RAG/vector database concepts and LLM API familiarity

Key Takeaways

  • Design document- and chunk-level metadata schemas that narrow the search space and reduce hallucinations
  • Extract breadcrumb headings, summaries, entities, dates and taxonomy tags automatically during ingestion
  • Combine structured self-query filters with hybrid vector search and payload indexing
  • Prevent outdated documents from polluting answers using version and temporal-validity metadata
  • Enforce document-level security by binding user identity (RBAC/ABAC groups) to vector pre-filters

Curriculum

The Role of Metadata in RAG and Agent Systems
  • Where naive vector similarity fails: temporal, structural and permission-bound queries
  • Document-level, chunk-level and tool-level metadata layers
Automated Metadata Extraction and Enrichment
  • Deterministic extraction (file properties, section breadcrumbs) and fast entity extraction (GLiNER)
  • Using structured-output LLMs to generate summaries, taxonomy tags and hypothetical questions
Payload Indexing, Pre-Filtering and Self-Querying
  • Filtered HNSW search performance in Qdrant, Milvus, pgvector and OpenSearch
  • Translating natural-language prompts into date, product and department filters via self-querying
Temporal Validity, Versioning and Verifiable Citations
  • Enforcing recency and policy precedence with valid_from, valid_to and is_latest fields
  • Preserving page numbers, section anchors and source URIs for verifiable citations
Security (ACL) Metadata and Agent Tool Routing
  • Syncing SharePoint/Drive/Confluence ACLs into chunk metadata for mandatory pre-filtering
  • Writing rich dataset and MCP tool metadata so agents route queries to the right source

Other Courses in This Category

  • Web Scraping Web Scraping

    Hands-on Web Scraping training covering scalable data extraction from static and dynamic web sources using BeautifulSoup, Scrapy, and Playwright, bypassing anti-bot defenses, LLM-powered data extraction, and data validation with Pandas.

    Data Science & AI-Ready Data Engineering Corporate, Individual
    Live Course
    13 modules
    View
  • Data Science Data Science

    End-to-end Data Science training spanning data analysis and machine learning with Python, Pandas, SQL, and Scikit-learn to deep learning with PyTorch/TensorFlow, LLMs, Apache Spark, and MLOps workflows.

    Data Science & AI-Ready Data Engineering Corporate, Individual
    Live Course
    18 modules
    View
  • Introduction to Data Science

    Data science lifecycle (CRISP-DM), exploratory data analysis, core machine learning concepts and model evaluation: a first end-to-end project for analysts and business teams who want to make data-driven decisions.

    Data Science & AI-Ready Data Engineering Corporate
    Live Course
    5 modules
    View
  • Python for Data Science

    Data loading, cleaning, transformation and visualisation with NumPy, pandas, Matplotlib/Seaborn and scikit-learn, for analysts and developers who want to write reusable analysis notebooks in Jupyter.

    Data Science & AI-Ready Data Engineering Corporate
    Live Course
    5 modules
    View
  • Statistics for Data Science

    Descriptive statistics, probability distributions, hypothesis testing, confidence intervals, A/B testing and regression, applied in Python so data scientists and analysts can draw statistically sound conclusions.

    Data Science & AI-Ready Data Engineering Corporate
    Live Course
    5 modules
    View
  • Advanced Data Science

    Gradient boosting (XGBoost, LightGBM), hyperparameter optimisation, time series forecasting, SHAP explainability and MLflow experiment tracking, for experienced data scientists taking models to production.

    Data Science & AI-Ready Data Engineering Corporate
    Live Course
    6 modules
    View

Get in touch