Model Online / On-Site
Duration
Modules 5 modules

Course Objective & Scope

When data outgrows what pandas can handle on a single machine, PySpark is the most practical way to carry Python skills into distributed processing. This course focuses on using Spark efficiently from within the Python ecosystem.

DataFrame transformations, Spark SQL, window functions, the difference between Python UDFs and pandas UDFs, the pandas API on Spark, broadcast joins and partition management, bottleneck analysis in the Spark UI and MERGE operations on Delta Lake are practised. Participants write and tune a PySpark job on an organisational dataset and package it as a reusable code template.

Who Should Attend?

  • Python developers and data engineers working with large datasets
  • Data scientists moving from pandas to distributed processing
  • Analytics teams using Databricks or EMR
  • Prerequisite: experience with Python and pandas

Key Takeaways

  • Write complex transformations with the PySpark DataFrame API
  • Gain performance by preferring built-in functions or pandas UDFs over Python UDFs
  • Identify shuffle and skew issues in the Spark UI
  • Perform upserts and time travel on Delta Lake tables
  • Move PySpark work from notebooks into production jobs

Curriculum

PySpark Architecture and SparkSession
  • Driver, executors and lazy evaluation
  • Transformations versus actions
The DataFrame API and Spark SQL
  • Select, filter, join and aggregation
  • Window functions
  • Complex types: arrays, structs, JSON
UDFs and the pandas API on Spark
  • Python UDFs versus pandas UDFs
  • Migrating with the pandas API on Spark
  • Arrow optimisation
Performance Optimisation
  • Broadcast joins, repartition and coalesce
  • Adaptive Query Execution and data skew
  • Analysis in the Spark UI
Delta Lake and Production Jobs
  • MERGE, schema evolution and time travel
  • Running with Databricks Jobs or spark-submit

Other Courses in This Category

  • Apache Kafka for Event Streaming

    Event streaming with Apache Kafka: topic, partition and consumer group design, producer/consumer development, Kafka Connect and Schema Registry, KRaft architecture, security and monitoring. For developers and ops teams.

    Big Data Corporate
    Live Course
    5 modules
    View
  • Apache Airflow: Workflow Orchestration for Data Pipelines

    Workflow orchestration for data and ML pipelines with Apache Airflow: DAG design with the TaskFlow API, dynamic task mapping, asset/dataset-driven scheduling, KubernetesExecutor, testing and production monitoring.

    Big Data Corporate
    Live Course
    5 modules
    View
  • Data Quality and Data Contracts for Reliable Data Platforms

    Data quality engineering and data contracts: schema, freshness and business rule validation with Great Expectations, Soda, dbt tests and OpenDataContractStandard; SLA management between producer and consumer teams.

    Big Data Corporate
    Live Course
    5 modules
    View
  • Observability for Data Platforms and Distributed Systems

    Observability for data pipelines and distributed systems: collecting metrics, logs, traces and data lineage with OpenTelemetry, Prometheus, Grafana and OpenLineage, anomaly detection and root-cause analysis.

    Big Data Corporate
    Live Course
    5 modules
    View
  • Big Data Big Data

    End-to-end cloud-based data engineering training covering relational and NoSQL databases, Data Lakehouse architectures, Apache Spark, Kafka, Airflow, and Terraform (IaC).

    Big Data Corporate, Individual
    Live Course
    11 modules
    View
  • Apache Spark Apache Spark

    Comprehensive big data processing training covering Apache Spark architecture, RDD and DataFrame APIs, Spark SQL, Structured Streaming, MLlib, GraphX, and distributed cluster performance tuning.

    Big Data Corporate, Individual
    Live Course
    10 modules
    View

Get in touch