Model Online / On-Site
Duration
Modules 5 modules

Course Objective & Scope

A silent schema change in a source application often breaks executive dashboards and machine learning models hours later. This programme moves data quality from after-the-fact clean-up to the ingestion boundary and formal agreements between teams.

Data quality dimensions, automated checks with dbt tests, SodaCL and Great Expectations, the write-audit-publish circuit breaker pattern and YAML-based data contract definitions are practised hands-on. Participants produce a sample data contract and a quality checklist for one of their own data flows.

Who Should Attend?

  • Data engineers and analytics engineers building pipelines and warehouses
  • Backend developers managing source system schemas
  • Data governance leads and Data Product Owners
  • Prerequisite: SQL and basic data pipeline experience

Key Takeaways

  • Define measurable quality rules across completeness, accuracy, consistency and freshness
  • Build automated quality gates with dbt tests, Soda or Great Expectations
  • Stop bad data from leaking downstream using the Write-Audit-Publish pattern
  • Author machine-readable data contracts between data producers and consumers
  • Add CI/CD validation steps that catch breaking schema changes early

Curriculum

Data Quality Dimensions and Failure Modes
  • Schema drift, null spikes, duplication and staleness
  • Data quality metrics and defining SLIs/SLOs
In-Pipeline Validation: dbt, Soda and Great Expectations
  • dbt generic and singular tests
  • Authoring rules with SodaCL and Great Expectations
  • The Write-Audit-Publish (circuit breaker) pattern
Data Contracts: Concepts and Standards
  • OpenDataContractStandard (ODCS) and YAML contract structure
  • Schema, semantics, quality rules and SLA clauses
  • Versioning and backward compatibility
CI/CD and Orchestration Integration
  • Schema and contract checks at pull-request time
  • Integrating with Airflow and the data catalogue
Enterprise Implementation Workshop
  • Drafting a data contract for a critical data asset
  • Incident response and ownership matrix

Other Courses in This Category

  • Observability for Data Platforms and Distributed Systems

    Observability for data pipelines and distributed systems: collecting metrics, logs, traces and data lineage with OpenTelemetry, Prometheus, Grafana and OpenLineage, anomaly detection and root-cause analysis.

    Big Data Corporate
    Live Course
    5 modules
    View
  • Big Data Big Data

    End-to-end cloud-based data engineering training covering relational and NoSQL databases, Data Lakehouse architectures, Apache Spark, Kafka, Airflow, and Terraform (IaC).

    Big Data Corporate, Individual
    Live Course
    11 modules
    View
  • Apache Spark Apache Spark

    Comprehensive big data processing training covering Apache Spark architecture, RDD and DataFrame APIs, Spark SQL, Structured Streaming, MLlib, GraphX, and distributed cluster performance tuning.

    Big Data Corporate, Individual
    Live Course
    10 modules
    View
  • Streaming Analytics: Real-Time Data Processing and Analysis

    Real-time analytics with Kafka, Apache Flink and Spark Structured Streaming: windowing, event time, watermarks, stateful processing and live dashboards. For data engineers and analytics teams.

    Big Data Corporate
    Live Course
    5 modules
    View
  • Hadoop, Hive and HBase: The Hadoop Ecosystem in Practice

    The Hadoop ecosystem with HDFS, YARN, Hive and HBase: distributed storage, SQL analytics in HiveQL, partitioning and ORC/Parquet tuning, low-latency access with HBase and migration paths to a modern lakehouse.

    Big Data Corporate
    Live Course
    5 modules
    View
  • Data Engineering Fundamentals

    An introduction to data engineering: ETL/ELT, data modelling, data warehouses and lakehouses, batch and streaming, orchestration and data quality, learned by designing an end-to-end sample data pipeline.

    Big Data Corporate
    Live Course
    5 modules
    View

Get in touch