Model Online / On-Site
Duration
Modules 5 modules

Course Objective & Scope

When analysts and AI developers spend weeks asking around just to find the right table and understand what its columns mean, data productivity grinds to a halt. This programme provides hands-on training in building a living, automatically updated data catalog paired with column-level data lineage.

DataHub and OpenMetadata architectures, harvesting metadata from warehouses, dbt and BI tools via connectors, capturing runtime lineage with OpenLineage, and setting up business glossaries and ownership are covered in practice. Participants build a working catalog prototype and a lineage/ownership model for one of their organisation's business domains.

Who Should Attend?

  • Data engineers building data platform and metadata infrastructure
  • Data governance specialists, data architects and data stewards
  • BI and analytics leads looking to accelerate self-service discovery and AI readiness
  • Prerequisite: SQL, data warehouse concepts and basic pipeline knowledge

Key Takeaways

  • Harvest metadata and query usage statistics automatically from databases, dbt, Airflow and BI tools
  • Construct end-to-end table- and column-level lineage graphs using OpenLineage and SQL parsers
  • Standardise business glossaries, domains, ownership and PII classifications in the catalog
  • Perform downstream impact analysis via lineage before deploying schema changes
  • Expose enriched catalog metadata so Text-to-SQL and enterprise AI assistants can locate authoritative tables

Curriculum

Modern Data Catalog Architecture (OpenMetadata & DataHub)
  • Metadata graphs, search indexing and storage architecture
  • Push- and pull-based metadata ingestion connectors
OpenLineage and Column-Level Lineage Extraction
  • Capturing runtime lineage events across Runs, Jobs and Datasets
  • Deriving column-level lineage from SQL query logs and dbt manifests
Business Glossary, Domains and the Ownership Model
  • Mapping technical tables and columns to business glossary terms
  • Domain-based governance, technical/business owner roles and certified datasets
Impact Analysis, Usage Analytics and Housekeeping
  • Identifying popular tables, unused orphan assets and expensive queries
  • Listing impacted dashboards, models and teams before rolling out schema changes
Catalog Integration for AI and RAG Systems
  • Serving schema and business context to LLMs and Text-to-SQL agents via catalog APIs
  • Tracking ML training datasets and RAG document collections inside the catalog

Other Courses in This Category

  • Data Observability: Monitoring, Drift Detection and Incident Management

    Detecting freshness, volume, schema and distribution anomalies across data pipelines, preventing alert fatigue, running root-cause analysis and managing data incidents for data and AI platform teams.

    Data Science & AI-Ready Data Engineering Corporate
    Live Course
    5 modules
    View
  • Multimodal Data Ingestion: Document, Audio and Image Parsing

    High-accuracy extraction of structured text and metadata from complex PDF tables, scanned forms, audio recordings and images using OCR, Docling, Whisper and vision-language models.

    Data Science & AI-Ready Data Engineering Corporate
    Live Course
    5 modules
    View
  • dbt and Analytics Engineering Transformation Workflows

    Modular SQL transformations with dbt Core/Cloud, staging-intermediate-mart layers, Jinja macros, incremental models, snapshots, testing and CI/CD workflows for data and analytics engineers.

    Data Science & AI-Ready Data Engineering Corporate
    Live Course
    5 modules
    View
  • Data and AI Pipeline Orchestration with Apache Airflow

    Authoring production DAGs with Apache Airflow: TaskFlow API, dynamic task mapping, dataset/asset-aware scheduling, isolated execution on Kubernetes, error handling and AI/RAG pipeline orchestration.

    Data Science & AI-Ready Data Engineering Corporate
    Live Course
    5 modules
    View
  • Dirty Data, Clean CV: Real-World ML, Interviews & Career Dirty Data, Clean CV

    Advanced workshop on applying regression to messy real-world data, presenting model metrics to non-technical stakeholders, acing ML technical interviews, and building an impactful Data Science CV and GitHub portfolio.

    Data Science & AI-Ready Data Engineering Individual Workshop
    Live Course
    4 modules
    View
  • Data Lakehouse Architecture with Apache Iceberg

    Warehouse reliability on object storage: Apache Iceberg table internals, metadata tree, hidden partitioning, time travel, REST Catalog, multi-engine querying (Trino, Spark, DuckDB) and table maintenance.

    Data Science & AI-Ready Data Engineering Corporate
    Live Course
    5 modules
    View

Get in touch