Model Online / On-Site
Duration
Modules 5 modules

Course Objective & Scope

Scripts that run in a notebook often break in production. This course aims to use Python for building data pipelines in a robust, testable and maintainable way.

Participants practise transformations on large tables with pandas and Polars, paginated extraction from REST APIs with requests/httpx, database connectivity via SQLAlchemy, efficient file handling with Parquet and Arrow, schema validation with pydantic, unit testing with pytest and structured logging. They build a packaged, tested ingestion module adapted to the organisation's scenario.

Who Should Attend?

  • Data engineers writing pipeline code
  • Data analysts and data scientists who want to move scripts into production
  • Backend developers working on data integration
  • Prerequisite: basic Python syntax

Key Takeaways

  • Write efficient transformations in pandas and Polars and compare their performance
  • Develop reliable ingestion code for API and database sources
  • Validate incoming data against a schema with pydantic
  • Write unit tests for pipeline functions with pytest
  • Structure code as a modular package with logging and retry logic

Curriculum

The Python Environment for Data Engineering
  • Virtual environments, uv/pip and dependency management
  • Project structure and configuration files
Transformation with pandas and Polars
  • Filtering, joins, groupby and window functions
  • Lazy evaluation and memory optimisation
  • File handling with Parquet and Arrow
Ingesting from APIs and Databases
  • REST APIs, pagination and rate limits
  • Reading and writing with SQLAlchemy
  • Incremental loading
Schema Validation and Error Handling
  • Data models with pydantic
  • Retry, timeout and exception strategies
  • Structured logging
Testing, Packaging and Production Readiness
  • Unit and integration tests with pytest
  • Packaging and containerising the code
  • Integrating with an orchestrator

Other Courses in This Category

  • PySpark: Distributed Data Processing with Python

    PySpark for Python developers: the DataFrame API, Spark SQL, UDFs and pandas API on Spark, join and shuffle optimisation, working with Delta Lake tables and running jobs on Databricks or EMR.

    Big Data Corporate
    Live Course
    5 modules
    View
  • Apache Kafka for Event Streaming

    Event streaming with Apache Kafka: topic, partition and consumer group design, producer/consumer development, Kafka Connect and Schema Registry, KRaft architecture, security and monitoring. For developers and ops teams.

    Big Data Corporate
    Live Course
    5 modules
    View
  • Apache Airflow: Workflow Orchestration for Data Pipelines

    Workflow orchestration for data and ML pipelines with Apache Airflow: DAG design with the TaskFlow API, dynamic task mapping, asset/dataset-driven scheduling, KubernetesExecutor, testing and production monitoring.

    Big Data Corporate
    Live Course
    5 modules
    View
  • Data Quality and Data Contracts for Reliable Data Platforms

    Data quality engineering and data contracts: schema, freshness and business rule validation with Great Expectations, Soda, dbt tests and OpenDataContractStandard; SLA management between producer and consumer teams.

    Big Data Corporate
    Live Course
    5 modules
    View
  • Observability for Data Platforms and Distributed Systems

    Observability for data pipelines and distributed systems: collecting metrics, logs, traces and data lineage with OpenTelemetry, Prometheus, Grafana and OpenLineage, anomaly detection and root-cause analysis.

    Big Data Corporate
    Live Course
    5 modules
    View
  • Big Data Big Data

    End-to-end cloud-based data engineering training covering relational and NoSQL databases, Data Lakehouse architectures, Apache Spark, Kafka, Airflow, and Terraform (IaC).

    Big Data Corporate, Individual
    Live Course
    11 modules
    View

Get in touch