Course Objective & Scope
While CPU-based pod autoscaling works for standard web apps, AI services face pre-allocated GPU memory, multi-gigabyte model downloads that cause minute-long cold starts and long-running inference calls that skew round-robin load balancers. This course covers the engineering practices required to scale ML and LLM services reliably and cost-effectively under heavy load.
Participants build autoscaling rules driven by queue depth and concurrency using Prometheus, HPA, KEDA and Knative/KServe, configure GPU time-slicing and MIG, slash pod startup times through model caching and orchestrate GPU node provisioning with Karpenter/Cluster Autoscaler. Teams leave with a scaling architecture and load-test report tailored to their environment.
Who Should Attend?
- MLOps and platform engineers operating model and LLM services on Kubernetes
- SRE and DevOps specialists responsible for the reliability of high-traffic AI systems
- Infrastructure architects designing cloud GPU clusters and scaling policies
- Prerequisite: hands-on experience with Kubernetes, Docker and basic model serving
Key Takeaways
- Author HPA and KEDA scaling rules driven by QPS, concurrency, queue depth and KV-cache usage rather than CPU
- Implement scale-to-zero for spiky or low-traffic models using Knative and KServe
- Slash cold-start times for large models using PVCs, local NVMe caching and container image streaming
- Share GPUs safely across multiple model pods with NVIDIA MIG and GPU time-slicing
- Configure least-request load balancing, backpressure, load shedding and GPU node provisioning with Karpenter
Curriculum
Vertical vs Horizontal Scaling and GPU Sharing
- Balancing intra-pod parallelism (workers/instances) against pod replica scaling
- Configuring Time-Slicing, MPS and Multi-Instance GPU (MIG) with the NVIDIA GPU Operator
Custom-Metric Pod Autoscaling: HPA, KEDA and Knative
- Scaling on DCGM GPU metrics, queue depth, in-flight requests and LLM KV-cache utilisation
- KEDA (Kafka/Redis queue and Prometheus triggers) and scale-to-zero with Knative/KServe
- Tuning stabilization windows and cooldowns to prevent thrashing
Reducing Cold Starts and Accelerating Large Model Startup
- Avoiding bloated images, using ReadWriteMany PVCs and node-local NVMe caching
- Parallel weight fetching, safetensors memory-mapping (
mmap) and warm spare overprovisioning
Intelligent Load Balancing, Backpressure and Resilience
- Least-Request and prefix/KV-cache-aware routing for variable-duration inference calls
- Max concurrency caps, bounded queues, circuit breakers and graceful load shedding (HTTP 429/503)
Node Autoscaling and Capacity Testing Workshop
- Managing GPU node pools and hybrid on-demand/spot capacity with Cluster Autoscaler and Karpenter
- Simulating traffic spikes, analysing scaling lag and building a production scaling checklist
Other Courses in This Category
-
The core steps for moving a trained model from notebook to production: serialization, serving via an API, container packaging, basic monitoring and rollback. For data scientists and developers new to model deployment.
- Live Course
- 5 modules
-
Fundamentals of REST and gRPC for model services: HTTP/JSON design, OpenAPI, Protocol Buffers, streaming RPCs and performance comparison. For developers who want to pick the right protocol and build their first service.
- Live Course
- 5 modules
-
Synchronous and asynchronous inference patterns: request-response, queue-based processing (Celery, Redis, Kafka), job polling, webhooks and async Python. For engineers designing long-running model calls that scale.
- Live Course
- 5 modules
-
A comparison of the three core inference modes: batch scoring with Spark and Airflow, REST-based real-time services and streaming inference with Kafka/Flink. For data and ML teams choosing the right mode for a business need.
- Live Course
- 5 modules
-
Saving models safely and portably: pickle/joblib risks, TorchScript, SavedModel, ONNX, safetensors and GGUF formats, packaging with MLflow and version compatibility. For ML engineers and data scientists.
- Live Course
- 5 modules
-
Model serving architecture patterns: embedded models, model-as-a-service, serverless, edge, multi-model and LLM gateway designs, and how platforms like KServe, Triton, Ray Serve and vLLM fit in. For architects and senior ML engineers.
- Live Course
- 5 modules
Get in touch
