Model Online / On-Site
Duration
Modules 5 modules

Course Objective & Scope

Kafka has become the enterprise event backbone for inter-service communication, CDC and real-time analytics. Yet a wrong partition count, poor offset management or uncontrolled schema changes can cause data loss and latency in production.

This programme covers Kafka architecture (brokers, KRaft, replication), topic and partition design, idempotent producers and transactions, consumer groups and rebalancing, Debezium CDC via Kafka Connect, Avro/Protobuf schema management with Schema Registry, and ACL and TLS security in hands-on labs. An event flow design and an operations checklist are prepared for the organisation's scenario.

Who Should Attend?

  • Backend developers building event-driven applications
  • Data engineers building pipelines on Kafka
  • Platform and operations teams running Kafka clusters
  • Prerequisite: basic distributed systems knowledge and Java, Python or Go

Key Takeaways

  • Set topic, partition and replication parameters for a given workload
  • Write producer and consumer code matching the required delivery guarantees
  • Stream database changes with Kafka Connect and Debezium
  • Manage backward-compatible schema evolution with Schema Registry
  • Diagnose problems by monitoring consumer lag and broker metrics

Curriculum

Kafka Architecture and KRaft
  • Brokers, topics, partitions and replication
  • Moving from ZooKeeper to KRaft
  • Log retention and compaction
Producer and Consumer Development
  • acks, idempotence and transactions
  • Consumer groups, offsets and rebalancing
  • At-least-once and exactly-once
Kafka Connect and CDC
  • Source and sink connectors
  • Database change streams with Debezium
Schema Management and Stream Processing
  • Schema Registry, Avro and Protobuf
  • Compatibility modes
  • Introduction to Kafka Streams and ksqlDB
Security, Monitoring and Operations
  • TLS, SASL and ACLs
  • Consumer lag and broker metrics
  • Capacity planning checklist

Other Courses in This Category

  • Apache Airflow: Workflow Orchestration for Data Pipelines

    Workflow orchestration for data and ML pipelines with Apache Airflow: DAG design with the TaskFlow API, dynamic task mapping, asset/dataset-driven scheduling, KubernetesExecutor, testing and production monitoring.

    Big Data Corporate
    Live Course
    5 modules
    View
  • Data Quality and Data Contracts for Reliable Data Platforms

    Data quality engineering and data contracts: schema, freshness and business rule validation with Great Expectations, Soda, dbt tests and OpenDataContractStandard; SLA management between producer and consumer teams.

    Big Data Corporate
    Live Course
    5 modules
    View
  • Observability for Data Platforms and Distributed Systems

    Observability for data pipelines and distributed systems: collecting metrics, logs, traces and data lineage with OpenTelemetry, Prometheus, Grafana and OpenLineage, anomaly detection and root-cause analysis.

    Big Data Corporate
    Live Course
    5 modules
    View
  • Big Data Big Data

    End-to-end cloud-based data engineering training covering relational and NoSQL databases, Data Lakehouse architectures, Apache Spark, Kafka, Airflow, and Terraform (IaC).

    Big Data Corporate, Individual
    Live Course
    11 modules
    View
  • Apache Spark Apache Spark

    Comprehensive big data processing training covering Apache Spark architecture, RDD and DataFrame APIs, Spark SQL, Structured Streaming, MLlib, GraphX, and distributed cluster performance tuning.

    Big Data Corporate, Individual
    Live Course
    10 modules
    View
  • Streaming Analytics: Real-Time Data Processing and Analysis

    Real-time analytics with Kafka, Apache Flink and Spark Structured Streaming: windowing, event time, watermarks, stateful processing and live dashboards. For data engineers and analytics teams.

    Big Data Corporate
    Live Course
    5 modules
    View

Get in touch