Course Objective & Scope
Simple green-or-red job status indicators do not explain why queries slow down or why data silently degrades. This course covers observability architectures that provide deep visibility across data platforms and distributed services.
Participants emit metrics, structured logs and distributed traces with OpenTelemetry, visualise them in Prometheus and Grafana, and track table- and column-level lineage with OpenLineage. Using a sample pipeline from the organisation, they design an end-to-end dashboard and SLI/SLO alert rules that avoid alert fatigue.
Who Should Attend?
- Data engineers operating data platforms and pipelines
- SRE, DevOps and platform teams monitoring distributed systems
- Software engineers responsible for production root-cause analysis
- Prerequisite: basic Python or application development and container familiarity
Key Takeaways
- Explain the difference between traditional monitoring and modern observability
- Emit metrics, logs and traces from applications and data jobs with the OpenTelemetry SDK
- Use OpenLineage to trace which downstream reports and models a failure affects
- Build actionable dashboards and SLO alerts with Prometheus and Grafana
- Speed up root-cause analysis for latency, volume anomalies and schema changes
Curriculum
Pillars of Observability: Metrics, Logs, Traces and Lineage
- Monitoring versus observability
- Combining system telemetry with data observability
Instrumentation with OpenTelemetry
- OTel Collector, SDKs and context propagation
- Correlating structured logs and traces
- Collecting signals from Spark, Airflow and Python services
Visualisation with Prometheus and Grafana
- Querying metrics with PromQL
- Golden signals and pipeline health dashboards
Data Observability and OpenLineage
- Freshness, volume, distribution, schema and lineage pillars
- OpenLineage/Marquez integration with Airflow, Spark and dbt
- Downstream impact analysis
SLI/SLO Design and Incident Response
- Error budgets and symptom-based alerting
- Reducing alert fatigue and writing runbooks
Other Courses in This Category
-
Big Data
End-to-end cloud-based data engineering training covering relational and NoSQL databases, Data Lakehouse architectures, Apache Spark, Kafka, Airflow, and Terraform (IaC).
- Live Course
- 11 modules
-
Apache Spark
Comprehensive big data processing training covering Apache Spark architecture, RDD and DataFrame APIs, Spark SQL, Structured Streaming, MLlib, GraphX, and distributed cluster performance tuning.
- Live Course
- 10 modules
-
Real-time analytics with Kafka, Apache Flink and Spark Structured Streaming: windowing, event time, watermarks, stateful processing and live dashboards. For data engineers and analytics teams.
- Live Course
- 5 modules
-
The Hadoop ecosystem with HDFS, YARN, Hive and HBase: distributed storage, SQL analytics in HiveQL, partitioning and ORC/Parquet tuning, low-latency access with HBase and migration paths to a modern lakehouse.
- Live Course
- 5 modules
-
An introduction to data engineering: ETL/ELT, data modelling, data warehouses and lakehouses, batch and streaming, orchestration and data quality, learned by designing an end-to-end sample data pipeline.
- Live Course
- 5 modules
-
Python for data engineers: data transformation with pandas and Polars, ingestion from APIs and databases, working with Parquet, testable and packaged pipeline code, logging and error handling.
- Live Course
- 5 modules
Get in touch
