Model Online / On-Site
Duration
Modules 5 modules

Course Objective & Scope

Data jobs chained together with blind cron schedules either run on incomplete upstream data when a source is late or require painful manual intervention when a step fails. This training teaches how to use modern Apache Airflow features to reliably orchestrate both classical ELT/dbt workflows and AI workloads such as document indexing and model training.

The TaskFlow API, idempotent task design and backfills, dynamic task mapping, deferrable operators and sensors, dataset/asset-driven scheduling, dependency isolation with KubernetesPodOperator and DAG testing are covered hands-on. Participants build a production-ready DAG template and operational checklist for one of their organisation's data or RAG indexing pipelines.

Who Should Attend?

  • Data engineers orchestrating ETL/ELT and data platform workflows
  • ML and AI engineers scheduling retraining, batch inference and RAG indexing pipelines
  • Platform and DevOps engineers operating Airflow infrastructure
  • Prerequisite: intermediate Python programming and basic Docker/SQL familiarity

Key Takeaways

  • Design atomic, idempotent Airflow tasks that can be safely retried and backfilled without duplicating data
  • Build clean, parallel workflows using the TaskFlow API, custom XCom backends and dynamic task mapping
  • Replace brittle clock-based schedules with event- and dataset/asset-aware DAG triggers
  • Prevent worker starvation and dependency conflicts using deferrable sensors, pools and KubernetesPodOperator
  • Harden pipelines with SLA callbacks, smart retries, OpenLineage tracking and automated DAG tests

Curriculum

Airflow Architecture and Modern DAG Design Principles
  • Scheduler, Webserver, Triggerer and Executors (Celery vs Kubernetes)
  • Idempotency, atomicity, logical dates (data_interval) and safe backfills
TaskFlow API, XComs and Dynamic Task Mapping
  • Writing clean Pythonic pipelines with @dag/@task and custom XCom backends
  • Spawning parallel runtime tasks dynamically with map and expand
Sensors, Deferrable Operators and Dataset-Aware Scheduling
  • Freeing worker slots with async Triggerer-backed deferrable operators
  • Chaining producer and consumer DAGs via Dataset/Asset updates
Environment Isolation, dbt and AI/RAG Workload Orchestration
  • Isolating heavy ML/Python dependencies with Virtualenv, Docker and KubernetesPodOperator
  • Orchestrating dbt projects (via Cosmos) and incremental RAG embedding pipelines
Error Handling, Observability and CI/CD Testing
  • Retries, exponential backoff, SLA callbacks and OpenLineage integration
  • Optimising DAG parse times, running DAG validation tests and deploying across environments

Other Courses in This Category

  • Dirty Data, Clean CV: Real-World ML, Interviews & Career Dirty Data, Clean CV

    Advanced workshop on applying regression to messy real-world data, presenting model metrics to non-technical stakeholders, acing ML technical interviews, and building an impactful Data Science CV and GitHub portfolio.

    Data Science & AI-Ready Data Engineering Individual Workshop
    Live Course
    4 modules
    View
  • Data Lakehouse Architecture with Apache Iceberg

    Warehouse reliability on object storage: Apache Iceberg table internals, metadata tree, hidden partitioning, time travel, REST Catalog, multi-engine querying (Trino, Spark, DuckDB) and table maintenance.

    Data Science & AI-Ready Data Engineering Corporate
    Live Course
    5 modules
    View
  • Semantic Layer Design for BI, Text-to-SQL and AI Agents

    Centralising enterprise metrics and dimensions with dbt Semantic Layer (MetricFlow) and Cube so BI tools, Text-to-SQL systems and analytical AI agents return consistent numbers.

    Data Science & AI-Ready Data Engineering Corporate
    Live Course
    5 modules
    View
  • Metadata Engineering for RAG and Autonomous Agents

    Metadata engineering to boost retrieval precision and security in RAG and agent systems: schema design, automated extraction, pre-filtering, temporal validity and ACL permission filtering.

    Data Science & AI-Ready Data Engineering Corporate
    Live Course
    5 modules
    View
  • Web Scraping Web Scraping

    Hands-on Web Scraping training covering scalable data extraction from static and dynamic web sources using BeautifulSoup, Scrapy, and Playwright, bypassing anti-bot defenses, LLM-powered data extraction, and data validation with Pandas.

    Data Science & AI-Ready Data Engineering Corporate, Individual
    Live Course
    13 modules
    View
  • Data Science Data Science

    End-to-end Data Science training spanning data analysis and machine learning with Python, Pandas, SQL, and Scikit-learn to deep learning with PyTorch/TensorFlow, LLMs, Apache Spark, and MLOps workflows.

    Data Science & AI-Ready Data Engineering Corporate, Individual
    Live Course
    18 modules
    View

Get in touch