Course Objective & Scope
Transformers sit at the core of modern AI across language, audio, vision and time series. Rather than treating the architecture as a black box, this course builds it layer by layer in PyTorch: scaled dot-product attention, causal and cross-attention masking, Multi-Head (MHA), Multi-Query (MQA) and Grouped-Query Attention (GQA), RoPE and ALiBi positional encodings, Pre-LN placement and SwiGLU feed-forward blocks.
When to choose encoder-only (BERT/DeBERTa), decoder-only (GPT/Llama/Qwen) or encoder-decoder (T5/BART) families is paired with hands-on KV-cache memory sizing and FlashAttention acceleration. Participants code a complete mini-Transformer and inspect production checkpoints in Hugging Face.
Who Should Attend?
- Data scientists and ML engineers working with LLMs, NLP and multimodal models
- Engineers handling fine-tuning, architectural adaptation and inference optimisation
- Technical teams seeking deep mastery of Transformer internals
- Prerequisite: PyTorch and foundational deep learning knowledge
Key Takeaways
- Implement self-attention, cross-attention, GQA and RoPE from scratch in PyTorch
- Select encoder-only, decoder-only or encoder-decoder backbones matched to the task
- Analyse how tokenisers (BPE, WordPiece, SentencePiece) affect domain accuracy and sequence length
- Calculate KV-cache memory footprints and reduce them with GQA and sliding-window attention
- Accelerate long-context training and inference using FlashAttention and PyTorch SDPA
Curriculum
The Attention Mechanism and Matrix Mechanics
- Query, Key, Value projections and scaled dot-product attention
- Padding masks, causal masks and cross-attention
The Modern Transformer Block: GQA, RoPE and SwiGLU
- Comparing MHA, MQA and Grouped-Query Attention (GQA)
- From sinusoidal encodings to RoPE, ALiBi and long-context scaling
Tokenisation and Architecture Families
- BPE, WordPiece, Unigram/SentencePiece and morphological fertility
- Matching Encoder (DeBERTa), Decoder (Llama/Qwen) and Seq2Seq (T5) to tasks
Building a Transformer from Scratch in PyTorch
- Coding Pre-RMSNorm, SwiGLU FFNs and residual pathways
- Training on a sequence task and inspecting attention patterns
KV Cache, FlashAttention and Long Context
- Sizing KV-cache memory during autoregressive decoding
- IO-aware FlashAttention and PyTorch SDPA acceleration
Other Courses in This Category
-
The Keras 3 and TensorFlow 2 ecosystem: Functional/Subclassing APIs, high-throughput tf.data pipelines, custom GradientTape loops, tf.distribute and enterprise deployment via SavedModel, TF Serving and TFLite.
- Live Course
- 5 modules
-
Quantisation (PTQ, QAT, INT8/INT4), structured pruning, knowledge distillation and deployment with ONNX Runtime, TensorRT, OpenVINO and LiteRT (TFLite) for lower cloud GPU costs and fast edge inference.
- Live Course
- 5 modules
-
Foundations and architectures of deep generative modelling: VAEs, GANs, Normalizing Flows, Denoising Diffusion (DDPM/Latent Diffusion), Flow Matching and autoregressive models for synthetic data and multimodal generation.
- Live Course
- 5 modules
-
MDPs, Q-Learning, DQN, Policy Gradients (PPO, SAC), contextual bandits and Offline RL using Gymnasium and Stable-Baselines3 for dynamic pricing, resource allocation and sequential control problems.
- Live Course
- 5 modules
-
Deep fraud models over transaction sequences, session clickstreams and payment networks: Transformer/RNN sequence encoders, autoencoder anomaly scores, GNNs for fraud rings and hybrid GBDT+DL architectures.
- Live Course
- 5 modules
-
Deep defect and fault detection across inspection images, acoustic/vibration signals and system logs using PatchCore/PaDiM, Anomalib, YOLO/segmentation, 1D-CNNs/autoencoders and normal-only training.
- Live Course
- 5 modules
Get in touch
