Model Online / On-Site
Duration
Modules 5 modules

Course Objective & Scope

A pipeline job finishing with a green status does not mean the data is right: null spikes, broken currency conversions or shifted distributions still slip silently into production tables. This focused programme equips data and analytics teams to apply software testing discipline to analytical tables and AI input datasets.

The distinction between transformation unit tests and data assertions, dbt unit and data tests, readable checks with SodaCL, profiling with Great Expectations and statistical drift assertions are practiced hands-on. Participants design a tiered testing strategy and a quarantine workflow for their own tables.

Who Should Attend?

  • Data engineers and analytics engineers building transformation and ETL/ELT pipelines
  • Data QA specialists responsible for warehouse and BI validation
  • ML engineers validating training and inference datasets
  • Prerequisite: strong SQL and basic Python knowledge

Key Takeaways

  • Write dbt unit tests that verify transformation logic against mock inputs
  • Automate schema, range, uniqueness and freshness checks with Soda Core and Great Expectations
  • Implement distribution drift, empty-text and outlier assertions for ML and RAG inputs
  • Calibrate test severity (warn vs fail) to avoid unnecessary pipeline halts
  • Build fast data quality gates that run on CI/CD pull requests

Curriculum

The Data Testing Pyramid: Unit Tests, Contracts and Assertions
  • Distinguishing logic unit tests from live data assertions
  • Deciding which checks belong in Bronze, Silver and Gold layers
Transformation and Unit Testing in dbt
  • Generic, singular and dbt-expectations test packages
  • Unit testing complex SQL/Jinja logic with mock inputs
Quality Gates with Soda Core and Great Expectations
  • Readable quality checks, freshness and row-count reconciliation with SodaCL
  • Automated data profiling and expectation suites in Great Expectations
Statistical and Semantic Checks for AI and ML Inputs
  • Feature distribution drift (PSI, KS test) and unseen categories
  • Validating document/chunk length, language, encoding and null embeddings
CI/CD Integration and Quarantine Patterns
  • Fast validation on sampled data during pull requests
  • Write-Audit-Publish (WAP), routing bad rows to quarantine and alerting

Other Courses in This Category

  • Data Contracts: Producer-Consumer Agreements and Schema Governance

    Practical training on securing data exchange between software producers and data/AI consumers using OpenDataContractStandard (ODCS), Schema Registry and automated CI/CD checks.

    Data Science & AI-Ready Data Engineering Corporate
    Live Course
    5 modules
    View
  • Data Lineage and Enterprise Data Catalogs

    Building an enterprise data catalog with OpenMetadata and DataHub, managing business glossaries, automating metadata ingestion and tracking column-level lineage with OpenLineage.

    Data Science & AI-Ready Data Engineering Corporate
    Live Course
    5 modules
    View
  • Data Observability: Monitoring, Drift Detection and Incident Management

    Detecting freshness, volume, schema and distribution anomalies across data pipelines, preventing alert fatigue, running root-cause analysis and managing data incidents for data and AI platform teams.

    Data Science & AI-Ready Data Engineering Corporate
    Live Course
    5 modules
    View
  • Multimodal Data Ingestion: Document, Audio and Image Parsing

    High-accuracy extraction of structured text and metadata from complex PDF tables, scanned forms, audio recordings and images using OCR, Docling, Whisper and vision-language models.

    Data Science & AI-Ready Data Engineering Corporate
    Live Course
    5 modules
    View
  • dbt and Analytics Engineering Transformation Workflows

    Modular SQL transformations with dbt Core/Cloud, staging-intermediate-mart layers, Jinja macros, incremental models, snapshots, testing and CI/CD workflows for data and analytics engineers.

    Data Science & AI-Ready Data Engineering Corporate
    Live Course
    5 modules
    View
  • Data and AI Pipeline Orchestration with Apache Airflow

    Authoring production DAGs with Apache Airflow: TaskFlow API, dynamic task mapping, dataset/asset-aware scheduling, isolated execution on Kubernetes, error handling and AI/RAG pipeline orchestration.

    Data Science & AI-Ready Data Engineering Corporate
    Live Course
    5 modules
    View

Get in touch