Model Online / On-Site
Duration
Modules 5 modules

Course Objective & Scope

Once an organisation has hundreds of tables and models, teams lose sight of which table is authoritative and which dashboards or AI models will break when a column changes. This programme teaches how to combine a data catalog, end-to-end lineage and data observability into a single operational control plane.

Setting up a catalog with OpenMetadata or DataHub, collecting column-level lineage via SQL AST parsing and OpenLineage, monitoring freshness/volume/distribution metrics and running root-cause analysis are practiced hands-on. Participants design a lineage/ownership model and an incident response workflow tailored to their own data landscape.

Who Should Attend?

  • Data platform and data engineering teams
  • Data governance specialists and data architects
  • AI engineers operating ML and RAG systems fed by multiple upstream sources
  • Prerequisite: SQL, modern warehouse/lakehouse concepts and basic pipeline knowledge

Key Takeaways

  • Catalog data assets with business glossary terms, tags, PII classification and ownership
  • Run upstream and downstream impact analysis using table- and column-level lineage
  • Monitor freshness, volume, schema and distribution metrics to catch anomalies before stakeholders do
  • Trace data incidents back to their root cause across the lineage graph
  • Design an internal data ownership and SLA alerting model

Curriculum

Modern Data Catalogs and Active Metadata
  • Moving from static data dictionaries to active metadata platforms (DataHub, OpenMetadata)
  • Business glossary, ownership, tiering and PII tagging
Table- and Column-Level Lineage Architecture
  • The OpenLineage standard, Airflow/dbt/Spark integration and SQL parsing
  • Column-level impact analysis and tracking downstream BI/AI consumers
The Five Pillars of Data Observability
  • Freshness, volume, schema, distribution and lineage metrics
  • Balancing explicit test rules with automated anomaly detection
Alert Management and Root-Cause Analysis (RCA)
  • Preventing alert fatigue, severity scoring and routing rules
  • Tracing broken pipelines and impacted reports/models across the lineage graph
Governance, Auditing and Integration with AI Systems
  • Privacy compliance (KVKK/GDPR), access auditing and retention policies
  • Source traceability for RAG indexes and ML feature tables

Other Courses in This Category

  • Unstructured Data Pipelines for Documents, Audio and Images

    Scalable data pipelines that turn PDFs, Office files, scanned documents, call-centre audio and images into parsed, chunked, metadata-enriched and indexed assets for RAG and multimodal AI.

    Data Science & AI-Ready Data Engineering Corporate
    Live Course
    5 modules
    View
  • The Modern Data Stack: dbt, Apache Airflow and Apache Iceberg

    Lakehouse storage on Apache Iceberg, modular tested transformations with dbt and orchestration with Apache Airflow, for engineers building scalable, open-standard data platforms.

    Data Science & AI-Ready Data Engineering Corporate
    Live Course
    5 modules
    View
  • AI-Ready Data Architecture

    Enterprise data architecture supporting analytics, classical ML, RAG and autonomous agents under one roof: designing lakehouse storage, semantic layers, feature stores, vector indexes and governance.

    Data Science & AI-Ready Data Engineering Corporate
    Live Course
    5 modules
    View
  • Automated Data Quality Testing for Analytics and AI

    Writing schema, business-rule, referential integrity and statistical distribution tests with dbt tests, Soda Core and Great Expectations to catch bad data in CI/CD and staging.

    Data Science & AI-Ready Data Engineering Corporate
    Live Course
    5 modules
    View
  • Data Contracts: Producer-Consumer Agreements and Schema Governance

    Practical training on securing data exchange between software producers and data/AI consumers using OpenDataContractStandard (ODCS), Schema Registry and automated CI/CD checks.

    Data Science & AI-Ready Data Engineering Corporate
    Live Course
    5 modules
    View
  • Data Lineage and Enterprise Data Catalogs

    Building an enterprise data catalog with OpenMetadata and DataHub, managing business glossaries, automating metadata ingestion and tracking column-level lineage with OpenLineage.

    Data Science & AI-Ready Data Engineering Corporate
    Live Course
    5 modules
    View

Get in touch