Model Online / On-Site
Duration
Modules 5 modules

Course Objective & Scope

An unannounced column change or a broken enum value in an operational database can silently corrupt downstream dashboards and AI models hours later. This training turns data quality from an after-the-fact cleanup chore into an engineering practice enforced in code between data producers and consumers.

Participants write schema, range, referential integrity and distribution tests using Great Expectations, Soda Core and dbt tests, then author YAML-based data contracts following the OpenDataContractStandard (ODCS) and enforce them in CI/CD and ingestion pipelines. Each team leaves with a working data contract example and test suite for one of the organisation's critical datasets.

Who Should Attend?

  • Data engineers and analytics engineers responsible for pipeline reliability
  • Software and backend developers producing application events and tables
  • Data architects defining platform and governance standards
  • Prerequisite: familiarity with SQL, Python and basic CI/CD workflows

Key Takeaways

  • Write automated checks for completeness, uniqueness, referential integrity, freshness and distribution drift
  • Prevent bad data from reaching production tables using the Write-Audit-Publish (WAP) pattern
  • Author a data contract covering schema, semantic rules, SLAs and ownership
  • Integrate breaking-change detection into CI/CD pipelines
  • Establish a clear accountability model between application producers and data consumers

Curriculum

Data Quality Dimensions and Failure Modes
  • Completeness, accuracy, consistency, timeliness and validity
  • Silent data bugs that break BI and AI systems
Automated Data Quality Tests (dbt, Soda, Great Expectations)
  • Writing schema, range, regex and custom SQL/Python assertions
  • Statistical thresholds and dynamic anomaly checks
Write-Audit-Publish (WAP) and Circuit Breakers
  • Staging validation and dead-letter / quarantine tables
  • Separating blocking failures from warning-level alerts
Data Contract Design and Standards
  • OpenDataContractStandard (ODCS), JSON Schema and Protobuf/Avro
  • Defining schema, business rules, SLAs, freshness and ownership
Contract Enforcement, Versioning and Team Workflow
  • Catching breaking changes in Git and CI/CD
  • Producer-consumer workflows and schema evolution

Other Courses in This Category

  • Data Lineage, Catalogs and Data Observability

    Discovering data assets, tracking column-level lineage and catching pipeline delays or distribution drift early using OpenLineage, DataHub/OpenMetadata and data observability practices.

    Data Science & AI-Ready Data Engineering Corporate
    Live Course
    5 modules
    View
  • Unstructured Data Pipelines for Documents, Audio and Images

    Scalable data pipelines that turn PDFs, Office files, scanned documents, call-centre audio and images into parsed, chunked, metadata-enriched and indexed assets for RAG and multimodal AI.

    Data Science & AI-Ready Data Engineering Corporate
    Live Course
    5 modules
    View
  • The Modern Data Stack: dbt, Apache Airflow and Apache Iceberg

    Lakehouse storage on Apache Iceberg, modular tested transformations with dbt and orchestration with Apache Airflow, for engineers building scalable, open-standard data platforms.

    Data Science & AI-Ready Data Engineering Corporate
    Live Course
    5 modules
    View
  • AI-Ready Data Architecture

    Enterprise data architecture supporting analytics, classical ML, RAG and autonomous agents under one roof: designing lakehouse storage, semantic layers, feature stores, vector indexes and governance.

    Data Science & AI-Ready Data Engineering Corporate
    Live Course
    5 modules
    View
  • Automated Data Quality Testing for Analytics and AI

    Writing schema, business-rule, referential integrity and statistical distribution tests with dbt tests, Soda Core and Great Expectations to catch bad data in CI/CD and staging.

    Data Science & AI-Ready Data Engineering Corporate
    Live Course
    5 modules
    View
  • Data Contracts: Producer-Consumer Agreements and Schema Governance

    Practical training on securing data exchange between software producers and data/AI consumers using OpenDataContractStandard (ODCS), Schema Registry and automated CI/CD checks.

    Data Science & AI-Ready Data Engineering Corporate
    Live Course
    5 modules
    View

Get in touch