Model Online / On-Site
Duration
Modules 5 modules

Course Objective & Scope

Reporting, analytics and AI projects all depend on reliable data pipelines. This programme introduces the core concepts of data engineering and the building blocks of a modern data platform in one coherent framework.

It covers ingestion from source systems, ETL and ELT approaches, star schemas and dimensional modelling, the differences between data warehouses, data lakes and lakehouses, batch and streaming processing, orchestration with Airflow and basic data quality checks. Participants design an end-to-end pipeline from one of the organisation's data sources and produce a reusable architecture checklist.

Who Should Attend?

  • Software developers and analysts moving into data engineering
  • Engineers new to data teams
  • IT and project managers running data platform initiatives
  • Prerequisite: basic SQL

Key Takeaways

  • Explain the layers and roles of a modern data platform
  • Choose between ETL and ELT for a given scenario
  • Build a simple dimensional model with fact and dimension tables
  • Distinguish batch from streaming requirements
  • Sketch the architecture of an end-to-end data pipeline

Curriculum

The Role of Data Engineering and the Data Lifecycle
  • Data engineer, analyst and data scientist roles
  • The data lifecycle from source to consumption
Ingestion: ETL, ELT and CDC
  • API, file and database sources
  • The Change Data Capture approach
  • An overview of tools such as Fivetran and Airbyte
Data Modelling and Storage
  • Star schemas and dimensional modelling
  • Data warehouse, data lake and lakehouse
  • Parquet and open table formats
Transformation and Orchestration
  • The transformation layer with SQL and dbt
  • Scheduling and dependency management with Airflow
Data Quality and End-to-End Design Workshop
  • Basic quality checks and monitoring
  • Pipeline design with the organisation's data
  • Architecture checklist

Other Courses in This Category

  • Python for Data Engineering

    Python for data engineers: data transformation with pandas and Polars, ingestion from APIs and databases, working with Parquet, testable and packaged pipeline code, logging and error handling.

    Big Data Corporate
    Live Course
    5 modules
    View
  • PySpark: Distributed Data Processing with Python

    PySpark for Python developers: the DataFrame API, Spark SQL, UDFs and pandas API on Spark, join and shuffle optimisation, working with Delta Lake tables and running jobs on Databricks or EMR.

    Big Data Corporate
    Live Course
    5 modules
    View
  • Apache Kafka for Event Streaming

    Event streaming with Apache Kafka: topic, partition and consumer group design, producer/consumer development, Kafka Connect and Schema Registry, KRaft architecture, security and monitoring. For developers and ops teams.

    Big Data Corporate
    Live Course
    5 modules
    View
  • Apache Airflow: Workflow Orchestration for Data Pipelines

    Workflow orchestration for data and ML pipelines with Apache Airflow: DAG design with the TaskFlow API, dynamic task mapping, asset/dataset-driven scheduling, KubernetesExecutor, testing and production monitoring.

    Big Data Corporate
    Live Course
    5 modules
    View
  • Data Quality and Data Contracts for Reliable Data Platforms

    Data quality engineering and data contracts: schema, freshness and business rule validation with Great Expectations, Soda, dbt tests and OpenDataContractStandard; SLA management between producer and consumer teams.

    Big Data Corporate
    Live Course
    5 modules
    View
  • Observability for Data Platforms and Distributed Systems

    Observability for data pipelines and distributed systems: collecting metrics, logs, traces and data lineage with OpenTelemetry, Prometheus, Grafana and OpenLineage, anomaly detection and root-cause analysis.

    Big Data Corporate
    Live Course
    5 modules
    View

Get in touch