Model Online / On-Site
Duration
Modules 5 modules

Course Objective & Scope

When model parameters, context lengths or dataset volumes outgrow a single GPU, training must shard model states (weights, gradients, optimizer states) and computation across multi-GPU and multi-node clusters. Configured poorly, distributed runs spend most of their time stalled on NCCL collective communications rather than matrix multiplication.

This advanced course covers GPU memory accounting, DeepSpeed ZeRO (Stages 1, 2, 3 and CPU/NVMe Offload), PyTorch FSDP / FSDP2, Context/Sequence Parallelism (Ulysses and Ring Attention), Hugging Face Accelerate orchestration and profiling Model FLOPs Utilization (MFU). Participants configure, profile and tune a multi-GPU LLM training pipeline.

Who Should Attend?

  • Machine learning and AI engineers training LLMs across multi-GPU or multi-node clusters
  • ML platform and infrastructure engineers managing GPU clusters on Kubernetes or Slurm
  • Researchers optimising throughput and memory for large-scale SFT, continual pre-training or RL runs
  • Prerequisite: solid PyTorch proficiency, Linux skills and single-GPU training experience

Key Takeaways

  • Calculate exact per-GPU memory budgets for any model size, batch size and sequence length
  • Configure DeepSpeed ZeRO Stages 1/2/3 and PyTorch FSDP2 sharding strategies for optimal throughput
  • Overlap backward-pass computation with NCCL all-gather and reduce-scatter collectives
  • Combine Data, Tensor, Pipeline and Context Parallelism across intra-node (NVLink) and inter-node networks
  • Diagnose communication bottlenecks, NCCL hangs and stragglers using PyTorch Profiler and distributed checkpointing

Curriculum

Distributed Training Mathematics and NCCL Collectives
  • The $16\Psi$ mixed-precision AdamW memory formula and activation memory scaling
  • NCCL collectives (all-reduce, all-gather, reduce-scatter) across NVLink and InfiniBand/RoCE
DeepSpeed ZeRO (Stages 1, 2, 3) and ZeRO-Offload
  • Throughput-versus-memory trade-offs across ZeRO-1 (optimizer), ZeRO-2 (+ gradients) and ZeRO-3 (+ weights)
  • When and how to use ZeRO-Offload to CPU RAM without starving GPU compute
  • Tuning communication bucket sizes and prefetching in DeepSpeed and Accelerate
PyTorch FSDP and FSDP2 (DTensor)
  • Transformer layer wrapping policies, FULL_SHARD, SHARD_GRAD_OP and node-aware HYBRID_SHARD
  • Per-parameter DTensor sharding in FSDP2, torch.compile integration and QLoRA + FSDP
  • Fast asynchronous checkpointing with torch.distributed.checkpoint (DCP)
3D Parallelism and Context Parallelism for Long Sequences
  • Combining intra-node Tensor Parallelism with inter-node Pipeline and Data Parallelism
  • Sharding long-sequence activations with DeepSpeed Ulysses and Ring Attention
Profiling, MFU Optimisation and Debugging
  • Inspecting compute-communication overlap in PyTorch Profiler and Perfetto traces
  • Maximising Model FLOPs Utilization (MFU), debugging NCCL hangs and elastic fault recovery

Other Courses in This Category

  • Reinforcement Learning for Reasoning Models (GRPO and RLVR)

    Advanced hands-on course on training DeepSeek-R1-style reasoning models using Reinforcement Learning with Verifiable Rewards (RLVR) and critic-free Group Relative Policy Optimization (GRPO).

    LLM Customisation & Model Adaptation Corporate
    Live Course
    5 modules
    View
  • Dataset Quality and Decontamination for LLM Training

    Course on auditing and cleaning LLM training datasets: exact and MinHash/semantic deduplication, train-test and benchmark decontamination, perplexity/judge quality filtering and PII scrubbing.

    LLM Customisation & Model Adaptation Corporate
    Live Course
    5 modules
    View
  • Prompting, RAG or Fine-Tuning: LLM Adaptation Decision Framework

    Hands-on course on choosing and combining context engineering, RAG and fine-tuning based on knowledge freshness, style/behaviour alignment, latency and TCO, producing an adaptation decision matrix.

    LLM Customisation & Model Adaptation Corporate
    Live Course
    5 modules
    View
  • Model and Data Readiness: Base Model Selection and Data Suitability

    Course on selecting open-weight or commercial base models by size, licence, tokenizer and hardware footprint, and auditing enterprise data volume, quality and privacy readiness before adaptation.

    LLM Customisation & Model Adaptation Corporate
    Live Course
    5 modules
    View
  • Quality, Cost and Risk Analysis in LLM Adaptation

    Course on modelling total cost of ownership (data curation, training and 24/7 serving), quality-versus-regression trade-offs and operational/regulatory risks for LLM customisation projects.

    LLM Customisation & Model Adaptation Corporate
    Live Course
    5 modules
    View
  • Baselines and Success Metrics: Pre-Adaptation Benchmarking for LLMs

    Course on building golden test sets, establishing zero/few-shot baselines, defining deterministic and LLM-as-a-judge metrics and setting acceptance thresholds before starting model adaptation.

    LLM Customisation & Model Adaptation Corporate
    Live Course
    5 modules
    View

Get in touch