Model Online / On-Site
Duration
Modules 5 modules

Course Objective & Scope

In many organisations critical data still lives on on-premise Hadoop clusters. This course is designed for teams that want to run existing Hadoop infrastructure efficiently, speed up Hive queries and use HBase for the right scenarios.

HDFS block structure and replication, YARN resource management, table design in HiveQL, partitioning and bucketing, ORC/Parquet formats and HBase row key design are practised hands-on. The final part uses a checklist to evaluate options for moving Hadoop workloads to Spark, Iceberg or Delta Lake based architectures.

Who Should Attend?

  • Data engineers and platform teams operating Hadoop clusters
  • Data analysts reporting on Hive
  • Architects planning a move from Hadoop to a modern data platform
  • Prerequisite: SQL and basic Linux command line

Key Takeaways

  • Explain the roles of HDFS and YARN and how they work together
  • Speed up Hive queries through partitioning, bucketing and file format choice
  • Design HBase row keys and column families that match access patterns
  • Choose correctly between Hive and HBase for a given use case
  • Evaluate modernisation options for existing Hadoop workloads

Curriculum

HDFS and YARN Architecture
  • NameNode, DataNode, blocks and replication
  • YARN ResourceManager and queue management
  • Essential HDFS commands
SQL Analytics with Hive
  • Managed and external tables, the Metastore
  • HiveQL queries and the Tez engine
  • Loading and transforming data
Hive Performance Tuning
  • Partitioning and bucketing strategies
  • ORC, Parquet and compression
  • Reading query plans with EXPLAIN
NoSQL Access with HBase
  • Regions, column families and versioning
  • Row key design and hotspot prevention
  • Get, Put and Scan operations
Hadoop Modernisation Roadmap
  • Spark, Iceberg and Delta Lake options
  • Assessing cloud or hybrid migration
  • Migration checklist

Other Courses in This Category

  • Data Engineering Fundamentals

    An introduction to data engineering: ETL/ELT, data modelling, data warehouses and lakehouses, batch and streaming, orchestration and data quality, learned by designing an end-to-end sample data pipeline.

    Big Data Corporate
    Live Course
    5 modules
    View
  • Python for Data Engineering

    Python for data engineers: data transformation with pandas and Polars, ingestion from APIs and databases, working with Parquet, testable and packaged pipeline code, logging and error handling.

    Big Data Corporate
    Live Course
    5 modules
    View
  • PySpark: Distributed Data Processing with Python

    PySpark for Python developers: the DataFrame API, Spark SQL, UDFs and pandas API on Spark, join and shuffle optimisation, working with Delta Lake tables and running jobs on Databricks or EMR.

    Big Data Corporate
    Live Course
    5 modules
    View
  • Apache Kafka for Event Streaming

    Event streaming with Apache Kafka: topic, partition and consumer group design, producer/consumer development, Kafka Connect and Schema Registry, KRaft architecture, security and monitoring. For developers and ops teams.

    Big Data Corporate
    Live Course
    5 modules
    View
  • Apache Airflow: Workflow Orchestration for Data Pipelines

    Workflow orchestration for data and ML pipelines with Apache Airflow: DAG design with the TaskFlow API, dynamic task mapping, asset/dataset-driven scheduling, KubernetesExecutor, testing and production monitoring.

    Big Data Corporate
    Live Course
    5 modules
    View
  • Data Quality and Data Contracts for Reliable Data Platforms

    Data quality engineering and data contracts: schema, freshness and business rule validation with Great Expectations, Soda, dbt tests and OpenDataContractStandard; SLA management between producer and consumer teams.

    Big Data Corporate
    Live Course
    5 modules
    View

Get in touch