Course Objective & Scope
For large deep learning and LLM workloads, the dominant expense is 24/7 inference GPUs rather than training, yet most production clusters utilise less than 20% of peak GPU compute due to memory-bandwidth bottlenecks. Starting from GPU hardware internals, this advanced course covers low-level acceleration techniques for deep learning and transformer inference.
The SRAM/HBM memory hierarchy, prefill (compute-bound) versus decode (memory-bound) phases, profiling with Nsight Systems and PyTorch Profiler, TensorRT / TensorRT-LLM compilation, FlashAttention, PagedAttention (KV cache), FP8/AWQ/GPTQ quantization and speculative decoding are practised hands-on. Participants run an optimisation study on their own models to improve tokens/sec and cost per million requests.
Who Should Attend?
- AI engineers optimising latency and GPU spend for LLM, VLM and deep learning services
- MLOps and AI platform engineers operating high-throughput inference infrastructure
- Systems developers working with vLLM, SGLang, TensorRT-LLM or custom PyTorch inference engines
- Prerequisite: PyTorch, transformer architectures and basic familiarity with GPU/CUDA execution
Key Takeaways
- Use the roofline model to determine whether a workload is compute-bound or memory-bandwidth-bound
- Pinpoint CPU-GPU synchronisation stalls and slow kernels using PyTorch Profiler and NVIDIA Nsight
- Size KV-cache memory footprints accurately and apply PagedAttention, prefix caching and KV-cache quantization
- Deploy FP8, INT8 (SmoothQuant) and INT4 (AWQ/GPTQ) quantization while benchmarking task accuracy
- Cut TTFT and inter-token latency (ITL) using tensor parallelism and speculative decoding on TensorRT-LLM, vLLM and SGLang
Curriculum
GPU Hardware Architecture and Inference Maths
- Streaming Multiprocessors (SMs), Tensor Cores, HBM versus SRAM bandwidth and arithmetic intensity (roofline)
- Contrasting bottlenecks of the prefill (TTFT) and autoregressive decode (ITL) phases
Profiling, Graph Compilation and Kernel Fusion
- Reading CUDA streams, kernel launch overhead and memory transfers with PyTorch Profiler and Nsight Systems
- Operator fusion and eliminating Python overhead with torch.compile, CUDA Graphs and TensorRT
Attention and KV-Cache Optimisations
- Memory arithmetic across Multi-Head (MHA), Grouped-Query (GQA) and Multi-Latent Attention (MLA)
- FlashAttention-2/3, PagedAttention and automatic prefix caching with RadixAttention (SGLang)
Advanced Quantization: FP8, SmoothQuant, AWQ and GPTQ
- Hardware trade-offs between weight-only (W4A16 - AWQ/GPTQ) and weight-plus-activation (W8A8 - FP8/SmoothQuant)
- Selecting calibration sets, measuring perplexity/task degradation and quantifying throughput gains
Multi-GPU Inference and Speculative Decoding
- Tensor Parallelism (TP) versus Pipeline Parallelism (PP) and NVLink/PCIe interconnect limits
- Accelerating autoregressive decoding 2-3x with draft models and Medusa/EAGLE speculative decoding
Other Courses in This Category
-
SageMaker Studio, Processing and Training Jobs, SageMaker Pipelines, Model Registry, Real-Time/Serverless/Async Inference, Model Monitor and IAM/VPC security, for teams running end-to-end ML on AWS.
- Live Course
- 5 modules
-
System and model telemetry with OpenTelemetry, Prometheus and Grafana, prediction logging, delayed ground-truth joining, live performance tracking and incident response, for MLOps and SRE teams.
- Live Course
- 5 modules
-
The maths of covariate, label and concept drift; PSI, KS, Wasserstein and MMD tests; label-free performance estimation with Evidently/NannyML and retraining strategies, for data scientists and MLOps teams.
- Live Course
- 5 modules
-
Feature Store architectures preventing training-serving skew and time leakage: offline (Parquet/Snowflake/BigQuery) and online (Redis) stores with Feast, point-in-time joins and streaming features, for data and MLOps teams.
- Live Course
- 5 modules
-
Pre-deployment fairness (Fairlearn), explainability (SHAP), probability calibration, uncertainty quantification (Conformal Prediction), guardrails, progressive rollouts and kill-switches, for ML and risk teams.
- Live Course
- 5 modules
-
Designing an enterprise AI platform unifying classical ML and GenAI/LLMs: compute/GPU orchestration, data and model layers, AI Gateways, multi-tenant isolation and reference architectures, for architects and platform leads.
- Live Course
- 5 modules
Get in touch
