Course Objective & Scope
As cron jobs and chained scripts grow, tracking dependencies, retrying failed steps and running historical backfills quickly become unmanageable. This course teaches how to orchestrate data and analytics pipelines reliably and at scale with Apache Airflow.
Participants write idempotent DAGs with the TaskFlow API, use sensors, hooks and provider packages, and set up dynamic task mapping and data-aware scheduling. A workshop tailored to the organisation's own ETL/ELT flow produces a tested DAG template and a production operations checklist.
Who Should Attend?
- Data engineers scheduling and operating data pipelines
- MLOps teams automating model training and batch inference workflows
- Platform engineers managing enterprise orchestration infrastructure
- Prerequisite: intermediate Python and basic SQL
Key Takeaways
- Develop clean, modular and idempotent DAGs using the TaskFlow API
- Build flexible workflows with dynamic task mapping and asset/dataset triggers
- Manage Connections, Variables and Secrets Backends securely
- Automate DAG integrity and unit tests with pytest
- Compare CeleryExecutor and KubernetesExecutor architectures and resolve bottlenecks
Curriculum
Airflow Architecture and DAG Lifecycle
- Scheduler, Webserver, Triggerer and Executor components
- DAGs, Tasks, Operators and runtime concepts
- Idempotency and backfill principles
TaskFlow API, XComs and Data-Aware Scheduling
- Modern DAG authoring with @dag and @task decorators
- Using XComs and custom XCom backends
- Dynamic task mapping and asset/dataset triggers
Providers, Sensors and External System Integration
- Database, cloud storage, dbt and Spark integrations
- Saving resources with deferrable operators and the Triggerer
- Connections, Variables and Vault/Secrets integration
Testing, CI/CD and Code Quality
- Preventing slow DAG parse times and top-level code pitfalls
- DAG validation and unit testing with pytest
- Git-based deployment and environment separation
Scaling and Observability in Production
- Using KubernetesExecutor and KubernetesPodOperator
- SLAs, callbacks, retries and alerting mechanisms
- Production checklist and troubleshooting
Other Courses in This Category
-
Data quality engineering and data contracts: schema, freshness and business rule validation with Great Expectations, Soda, dbt tests and OpenDataContractStandard; SLA management between producer and consumer teams.
- Live Course
- 5 modules
-
Observability for data pipelines and distributed systems: collecting metrics, logs, traces and data lineage with OpenTelemetry, Prometheus, Grafana and OpenLineage, anomaly detection and root-cause analysis.
- Live Course
- 5 modules
-
Big Data
End-to-end cloud-based data engineering training covering relational and NoSQL databases, Data Lakehouse architectures, Apache Spark, Kafka, Airflow, and Terraform (IaC).
- Live Course
- 11 modules
-
Apache Spark
Comprehensive big data processing training covering Apache Spark architecture, RDD and DataFrame APIs, Spark SQL, Structured Streaming, MLlib, GraphX, and distributed cluster performance tuning.
- Live Course
- 10 modules
-
Real-time analytics with Kafka, Apache Flink and Spark Structured Streaming: windowing, event time, watermarks, stateful processing and live dashboards. For data engineers and analytics teams.
- Live Course
- 5 modules
-
The Hadoop ecosystem with HDFS, YARN, Hive and HBase: distributed storage, SQL analytics in HiveQL, partitioning and ORC/Parquet tuning, low-latency access with HBase and migration paths to a modern lakehouse.
- Live Course
- 5 modules
Get in touch
