Course Objective & Scope
Standard training scripts hit memory ceilings, unstable gradients and poor GPU utilisation when scaled to large datasets, deep architectures and scarce labels. This advanced programme covers automatic mixed precision (AMP BF16/FP16), torch.compile, gradient accumulation and clipping, distributed training with PyTorch DDP/FSDP, cosine warmup schedules, and self-supervised and contrastive learning (InfoNCE, Triplet/ArcFace).
In-domain pretraining on unlabelled enterprise data, knowledge distillation from teacher to student networks and bottleneck analysis with the PyTorch Profiler are practised hands-on. Participants leave with an accelerated, reusable multi-GPU training template.
Who Should Attend?
- Senior data scientists and ML engineers training deep models for production
- AI platform engineers optimising large-scale training and GPU utilisation
- R&D teams working on vision, audio, text or multimodal representation learning
- Prerequisite: hands-on experience training neural networks in PyTorch
Key Takeaways
- Maximise GPU memory efficiency with AMP (BF16/FP16), gradient accumulation and activation checkpointing
- Scale training across multiple GPUs with PyTorch DDP and FSDP
- Learn high-quality representations from unlabelled data via contrastive and self-supervised pretraining
- Compress heavy teacher models into fast student networks with knowledge distillation
- Eliminate data-loading, host-to-device and kernel bottlenecks using the PyTorch Profiler
Curriculum
Training Dynamics and Numerical Stability
- Warmup, cosine annealing, 1cycle and Lion/AdamW tuning
- Gradient clipping, diagnosing loss spikes and spectral/RMSNorm
Mixed Precision, Memory and Throughput Optimisation
- FP16/BF16 AMP, GradScaler and activation checkpointing
- torch.compile, pinned memory and the PyTorch Profiler
Multi-GPU and Distributed Training
- Why DataParallel fails and how DistributedDataParallel (DDP) works
- ZeRO stages and Fully Sharded Data Parallel (FSDP)
Self-Supervised and Metric Learning
- Embedding spaces with contrastive loss, InfoNCE, Triplet and ArcFace
- Masked pretraining and label-efficient fine-tuning strategies
Knowledge Distillation and Multi-Task Learning
- Logit and intermediate-feature distillation into student models
- Multi-task heads, loss balancing and a reusable training template
Other Courses in This Category
-
From ResNet and ConvNeXt to Vision Transformers (ViT), Swin, hybrid backbones, State Space Models (Mamba), Mixture-of-Experts (MoE) and multimodal fusion: selecting and designing architectures for your problem and hardware.
- Live Course
- 5 modules
-
Self-attention mechanics, Multi-Head and Grouped-Query Attention, RoPE positional encodings, encoder/decoder/encoder-decoder families, KV caching and FlashAttention, implemented from scratch in PyTorch and with Hugging Face.
- Live Course
- 5 modules
-
The Keras 3 and TensorFlow 2 ecosystem: Functional/Subclassing APIs, high-throughput tf.data pipelines, custom GradientTape loops, tf.distribute and enterprise deployment via SavedModel, TF Serving and TFLite.
- Live Course
- 5 modules
-
Quantisation (PTQ, QAT, INT8/INT4), structured pruning, knowledge distillation and deployment with ONNX Runtime, TensorRT, OpenVINO and LiteRT (TFLite) for lower cloud GPU costs and fast edge inference.
- Live Course
- 5 modules
-
Foundations and architectures of deep generative modelling: VAEs, GANs, Normalizing Flows, Denoising Diffusion (DDPM/Latent Diffusion), Flow Matching and autoregressive models for synthetic data and multimodal generation.
- Live Course
- 5 modules
-
MDPs, Q-Learning, DQN, Policy Gradients (PPO, SAC), contextual bandits and Offline RL using Gymnasium and Stable-Baselines3 for dynamic pricing, resource allocation and sequential control problems.
- Live Course
- 5 modules
Get in touch
