Model Online / On-Site
Duration
Modules 5 modules

Course Objective & Scope

Storing raw Parquet files in object-storage folders breaks down at enterprise scale due to concurrent write conflicts, brittle hive-style partitions and costly full-directory listings. This programme dives into the internals of the Apache Iceberg open table format and shows how to build a multi-engine, vendor-neutral data lakehouse.

The catalog, metadata file, manifest list, manifest file and Parquet layers, ACID transactions, hidden partitioning and partition evolution, Copy-on-Write vs Merge-on-Read updates, REST Catalog implementations (Polaris, Nessie, Lakekeeper) and compaction routines are practiced hands-on. Participants create an Iceberg table design and maintenance playbook tailored to their workloads.

Who Should Attend?

  • Data and platform engineers building data lake and lakehouse infrastructure
  • Data architects designing storage for large-scale analytics and AI workloads
  • Big data developers tuning Spark, Trino or Flink pipelines
  • Prerequisite: SQL, columnar storage concepts (Parquet) and basic distributed data processing knowledge

Key Takeaways

  • Explain how Iceberg's metadata tree (manifest lists, manifests, data and delete files) accelerates query planning
  • Apply in-place schema evolution and partition evolution without rewriting historical data
  • Choose between Copy-on-Write (CoW) and Merge-on-Read (MoR) based on update/delete frequency and read SLAs
  • Use time travel, branching and tagging for reproducible ML training snapshots and Write-Audit-Publish (WAP) workflows
  • Configure compaction, sort orders/Z-ordering and snapshot expiration to eliminate the small-file problem

Curriculum

Apache Iceberg Internals and the Metadata Tree
  • Limitations of Hive table layouts and Iceberg's three-tier metadata tree
  • Snapshot isolation, optimistic concurrency and file pruning via column stats
Schema Evolution, Hidden Partitioning and Row-Level Updates
  • Column-ID schema evolution and error-proof hidden partitioning
  • Copy-on-Write (CoW) vs Merge-on-Read (positional and equality delete files)
Catalog Ecosystem and Multi-Engine Interoperability
  • Iceberg REST Catalog standard, Apache Polaris, Project Nessie and Glue/Hive compared
  • Reading and writing the same tables across Spark, Trino, Flink and DuckDB/PyIceberg
Time Travel, Branching and AI/ML Dataset Reproducibility
  • Querying historical snapshots to reproduce exact ML training datasets
  • Table-level branches and tags for isolated Write-Audit-Publish (WAP) validation
Performance Tuning and Table Maintenance Operations
  • Bin-packing, sort and Z-order compaction to boost query performance
  • Automating expire_snapshots, remove_orphan_files and rewrite_manifests

Other Courses in This Category

  • Semantic Layer Design for BI, Text-to-SQL and AI Agents

    Centralising enterprise metrics and dimensions with dbt Semantic Layer (MetricFlow) and Cube so BI tools, Text-to-SQL systems and analytical AI agents return consistent numbers.

    Data Science & AI-Ready Data Engineering Corporate
    Live Course
    5 modules
    View
  • Metadata Engineering for RAG and Autonomous Agents

    Metadata engineering to boost retrieval precision and security in RAG and agent systems: schema design, automated extraction, pre-filtering, temporal validity and ACL permission filtering.

    Data Science & AI-Ready Data Engineering Corporate
    Live Course
    5 modules
    View
  • Web Scraping Web Scraping

    Hands-on Web Scraping training covering scalable data extraction from static and dynamic web sources using BeautifulSoup, Scrapy, and Playwright, bypassing anti-bot defenses, LLM-powered data extraction, and data validation with Pandas.

    Data Science & AI-Ready Data Engineering Corporate, Individual
    Live Course
    13 modules
    View
  • Data Science Data Science

    End-to-end Data Science training spanning data analysis and machine learning with Python, Pandas, SQL, and Scikit-learn to deep learning with PyTorch/TensorFlow, LLMs, Apache Spark, and MLOps workflows.

    Data Science & AI-Ready Data Engineering Corporate, Individual
    Live Course
    18 modules
    View
  • Introduction to Data Science

    Data science lifecycle (CRISP-DM), exploratory data analysis, core machine learning concepts and model evaluation: a first end-to-end project for analysts and business teams who want to make data-driven decisions.

    Data Science & AI-Ready Data Engineering Corporate
    Live Course
    5 modules
    View
  • Python for Data Science

    Data loading, cleaning, transformation and visualisation with NumPy, pandas, Matplotlib/Seaborn and scikit-learn, for analysts and developers who want to write reusable analysis notebooks in Jupyter.

    Data Science & AI-Ready Data Engineering Corporate
    Live Course
    5 modules
    View

Get in touch