Course Objective & Scope
Running AI workloads on Kubernetes is very different from deploying stateless web microservices: 10 GB CUDA container images, idle expensive GPUs, slow model-weight downloads and autoscaling on request queues rather than CPU all require specialised patterns. This course covers the full stack from container optimisation to orchestrating enterprise AI workloads on Kubernetes.
Multi-stage CUDA Dockerfiles, NVIDIA GPU Operator, GPU sharing via MIG and time-slicing, autoscaling inference services with KServe and KEDA, and multi-GPU training jobs with the Kubeflow Training Operator are practised hands-on. Participants leave with production-ready Helm/Kustomize templates and a GPU cluster scheduling blueprint.
Who Should Attend?
- MLOps, DevOps and SRE engineers operating AI and data science workloads on Kubernetes
- Platform engineers managing on-premises or cloud (EKS, GKE, AKS) GPU clusters
- AI infrastructure specialists deploying large models as scalable services
- Prerequisite: Working knowledge of Docker and core Kubernetes concepts (Pods, Deployments, Services)
Key Takeaways
- Shrink and accelerate PyTorch/CUDA container builds using multi-stage Dockerfiles and layer caching
- Share GPUs safely across multiple pods using the NVIDIA GPU Operator, Time-Slicing and Multi-Instance GPU (MIG)
- Decouple multi-gigabyte model weights from container images via PVCs, storage initialisers and local NVMe caches
- Configure request-queue and custom GPU-metric autoscaling (including scale-to-zero) with KServe and KEDA
- Schedule and prioritise batch training versus real-time inference jobs using Kubeflow Training Operator and Kueue
Curriculum
Advanced Container Engineering for AI/ML Workloads
- Choosing CUDA base images (runtime vs devel), multi-stage builds, uv/pip cache mounts and image slimming
- Decoupling model weights from container images, rootless execution and CUDA compatibility checks
GPU Management and Sharing on Kubernetes
- Deploying the NVIDIA GPU Operator, driver/toolkit lifecycle and GPU telemetry with DCGM Exporter
- GPU Time-Slicing, Multi-Instance GPU (MIG) profiles, node selectors, taints/tolerations and topology awareness
Scalable Model Serving with KServe (InferenceService)
- KServe architecture: Storage Initializers (S3/PVC), Predictors, Transformers and Explainers
- Canary traffic splitting, chaining models with InferenceGraph and local model caching
Autoscaling (KEDA/Knative) and Cold-Start Optimisation
- Why CPU HPA fails for ML: scaling on request concurrency, queue depth and DCGM GPU metrics with KEDA
- Tuning startup/readiness probes for large models and accelerating weight loading via node-local NVMe caches
Distributed Training and Job Queuing on Kubernetes (Kueue)
- Running multi-node DDP training jobs with the Kubeflow Training Operator (PyTorchJob)
- Managing team quotas, batch queuing, spot nodes and training-vs-inference preemption with Kueue
Other Courses in This Category
-
GPU memory/compute architecture, profiling with Nsight Systems, TensorRT, FlashAttention, KV cache management, AWQ/GPTQ/FP8 quantization and speculative decoding, for engineers cutting deep learning and LLM inference costs.
- Live Course
- 5 modules
-
SageMaker Studio, Processing and Training Jobs, SageMaker Pipelines, Model Registry, Real-Time/Serverless/Async Inference, Model Monitor and IAM/VPC security, for teams running end-to-end ML on AWS.
- Live Course
- 5 modules
-
System and model telemetry with OpenTelemetry, Prometheus and Grafana, prediction logging, delayed ground-truth joining, live performance tracking and incident response, for MLOps and SRE teams.
- Live Course
- 5 modules
-
The maths of covariate, label and concept drift; PSI, KS, Wasserstein and MMD tests; label-free performance estimation with Evidently/NannyML and retraining strategies, for data scientists and MLOps teams.
- Live Course
- 5 modules
-
Feature Store architectures preventing training-serving skew and time leakage: offline (Parquet/Snowflake/BigQuery) and online (Redis) stores with Feast, point-in-time joins and streaming features, for data and MLOps teams.
- Live Course
- 5 modules
-
Pre-deployment fairness (Fairlearn), explainability (SHAP), probability calibration, uncertainty quantification (Conformal Prediction), guardrails, progressive rollouts and kill-switches, for ML and risk teams.
- Live Course
- 5 modules
Get in touch
