Model Online / On-Site
Duration
Modules 5 modules

Course Objective & Scope

Vendor lock-in, tangled stored procedures and fragile cron scripts cannot keep pace with modern analytics and AI workloads. This programme teaches how to combine the open table format Apache Iceberg, the transformation layer dbt and the orchestrator Apache Airflow into an end-to-end modern data platform.

Hands-on labs cover creating Iceberg tables on object storage, querying them with Trino/Spark or DuckDB, building tested Bronze-Silver-Gold models in dbt, and orchestrating the entire pipeline with Airflow DAGs. Participants translate one of their organisation's data workflows into a working reference architecture built on this stack.

Who Should Attend?

  • Data engineers modernising data warehouse and data lake infrastructure
  • Analytics engineers applying software engineering practices to SQL transformations
  • Platform engineers and architects responsible for enterprise data platforms
  • Prerequisite: strong SQL skills, basic Python and familiarity with Git

Key Takeaways

  • Define clear responsibilities across Apache Iceberg, dbt and Airflow in a lakehouse architecture
  • Use ACID transactions, time travel, schema evolution and hidden partitioning on Iceberg tables
  • Build a modular dbt project with incremental models, macros, tests and documentation
  • Orchestrate ingestion, dbt runs and quality checks with dependency-aware Airflow DAGs
  • Plan table maintenance (compaction, snapshot expiration), cost and performance practices

Curriculum

Modern Lakehouse Architecture and Role of Each Layer
  • Decoupling storage, open table formats, query engines, transformation and orchestration
  • End-to-end design of Medallion (Bronze-Silver-Gold) layers
The Open Table Layer with Apache Iceberg
  • Metadata tree, snapshots, time travel and schema evolution
  • Hidden partitioning, file sizing and catalog integration
Modular Transformation and Testing with dbt
  • Staging, intermediate and mart models; view, table and incremental materializations
  • Source freshness checks, schema tests and automated lineage
End-to-End Orchestration with Apache Airflow
  • TaskFlow API, sensors and dataset-aware scheduling
  • Triggering dbt models and Iceberg maintenance jobs from Airflow
CI/CD, Environment Management and Operations
  • Dev/staging/prod separation, Slim CI and isolated audit schemas (WAP)
  • Compaction, orphan file cleanup and cost/performance monitoring

Other Courses in This Category

  • AI-Ready Data Architecture

    Enterprise data architecture supporting analytics, classical ML, RAG and autonomous agents under one roof: designing lakehouse storage, semantic layers, feature stores, vector indexes and governance.

    Data Science & AI-Ready Data Engineering Corporate
    Live Course
    5 modules
    View
  • Automated Data Quality Testing for Analytics and AI

    Writing schema, business-rule, referential integrity and statistical distribution tests with dbt tests, Soda Core and Great Expectations to catch bad data in CI/CD and staging.

    Data Science & AI-Ready Data Engineering Corporate
    Live Course
    5 modules
    View
  • Data Contracts: Producer-Consumer Agreements and Schema Governance

    Practical training on securing data exchange between software producers and data/AI consumers using OpenDataContractStandard (ODCS), Schema Registry and automated CI/CD checks.

    Data Science & AI-Ready Data Engineering Corporate
    Live Course
    5 modules
    View
  • Data Lineage and Enterprise Data Catalogs

    Building an enterprise data catalog with OpenMetadata and DataHub, managing business glossaries, automating metadata ingestion and tracking column-level lineage with OpenLineage.

    Data Science & AI-Ready Data Engineering Corporate
    Live Course
    5 modules
    View
  • Data Observability: Monitoring, Drift Detection and Incident Management

    Detecting freshness, volume, schema and distribution anomalies across data pipelines, preventing alert fatigue, running root-cause analysis and managing data incidents for data and AI platform teams.

    Data Science & AI-Ready Data Engineering Corporate
    Live Course
    5 modules
    View
  • Multimodal Data Ingestion: Document, Audio and Image Parsing

    High-accuracy extraction of structured text and metadata from complex PDF tables, scanned forms, audio recordings and images using OCR, Docling, Whisper and vision-language models.

    Data Science & AI-Ready Data Engineering Corporate
    Live Course
    5 modules
    View

Get in touch