Model Online / On-Site
Duration
Modules 5 modules

Course Objective & Scope

Nightly batch jobs cannot support decisions that must be made in seconds, such as fraud detection, inventory tracking or user behaviour monitoring. This programme covers streaming analytics architectures that turn event streams into metrics and actions almost instantly.

Participants process events arriving via Kafka with Flink and Spark Structured Streaming, applying tumbling, sliding and session windows, the difference between event time and processing time, and watermarks for late data. A case built around the organisation's own event types ends with a working streaming pipeline and a live dashboard.

Who Should Attend?

  • Data engineers building or planning real-time pipelines
  • Analytics and BI teams producing live KPIs and alerts
  • Software architects designing event-driven systems
  • Prerequisite: SQL and basic Python or Java/Scala

Key Takeaways

  • Choose correctly between batch, micro-batch and true real-time processing
  • Handle late-arriving data using event time and watermarks
  • Write window-based aggregations and stream-stream join queries
  • Build fault-tolerant streaming jobs with checkpoints and exactly-once semantics
  • Connect streaming results to live dashboards and alert rules

Curriculum

Streaming Architecture and Core Concepts
  • Events, streams, topics and partitions
  • Lambda and Kappa architectures
  • Batch versus streaming
Time, Windows and Watermarks
  • Event time and processing time
  • Tumbling, sliding and session windows
  • Late data handling with watermarks
Hands-On with Spark Structured Streaming
  • Reading from Kafka and output modes
  • Stateful aggregation and stream-stream joins
  • Checkpoint and trigger settings
Low-Latency Processing with Apache Flink
  • DataStream API and Flink SQL
  • Keyed state and savepoints
  • Exactly-once guarantees
Real-Time Dashboards and Alerting
  • Writing results to ClickHouse, Druid or Elasticsearch
  • Live visualisation with Grafana
  • Threshold and anomaly-based alert rules

Other Courses in This Category

  • Hadoop, Hive and HBase: The Hadoop Ecosystem in Practice

    The Hadoop ecosystem with HDFS, YARN, Hive and HBase: distributed storage, SQL analytics in HiveQL, partitioning and ORC/Parquet tuning, low-latency access with HBase and migration paths to a modern lakehouse.

    Big Data Corporate
    Live Course
    5 modules
    View
  • Data Engineering Fundamentals

    An introduction to data engineering: ETL/ELT, data modelling, data warehouses and lakehouses, batch and streaming, orchestration and data quality, learned by designing an end-to-end sample data pipeline.

    Big Data Corporate
    Live Course
    5 modules
    View
  • Python for Data Engineering

    Python for data engineers: data transformation with pandas and Polars, ingestion from APIs and databases, working with Parquet, testable and packaged pipeline code, logging and error handling.

    Big Data Corporate
    Live Course
    5 modules
    View
  • PySpark: Distributed Data Processing with Python

    PySpark for Python developers: the DataFrame API, Spark SQL, UDFs and pandas API on Spark, join and shuffle optimisation, working with Delta Lake tables and running jobs on Databricks or EMR.

    Big Data Corporate
    Live Course
    5 modules
    View
  • Apache Kafka for Event Streaming

    Event streaming with Apache Kafka: topic, partition and consumer group design, producer/consumer development, Kafka Connect and Schema Registry, KRaft architecture, security and monitoring. For developers and ops teams.

    Big Data Corporate
    Live Course
    5 modules
    View
  • Apache Airflow: Workflow Orchestration for Data Pipelines

    Workflow orchestration for data and ML pipelines with Apache Airflow: DAG design with the TaskFlow API, dynamic task mapping, asset/dataset-driven scheduling, KubernetesExecutor, testing and production monitoring.

    Big Data Corporate
    Live Course
    5 modules
    View

Get in touch