Course Objective & Scope
Wrapping a model in a basic Python web handler works for low-traffic internal tools, but under hundreds of requests per second the Python GIL, unbatched GPU calls and memory spikes quickly bring the service down. This advanced course covers production serving architectures that deliver high throughput and predictable p95/p99 tail latency across classical ML, deep learning and LLMs.
Online (REST/gRPC), asynchronous queue-based (Kafka/Celery) and batch inference patterns are compared hands-on using NVIDIA Triton Inference Server, BentoML, Ray Serve and vLLM. Dynamic batching, concurrent model execution, multi-model DAG pipelines and load testing with Locust and Perf Analyzer are practised to produce a tailored serving architecture.
Who Should Attend?
- MLOps, AI and backend engineers building high-traffic prediction and inference services
- Platform teams tasked with improving GPU/CPU utilisation and cutting serving costs
- Architects responsible for real-time recommendation, fraud detection, vision or LLM endpoints
- Prerequisite: Python, Docker and experience building REST APIs
Key Takeaways
- Choose between online (REST/gRPC), streaming and batch inference patterns based on business SLAs
- Multiply GPU throughput using dynamic batching on NVIDIA Triton Inference Server, BentoML and Ray Serve
- Compose preprocessing, multiple models and post-processing into a single inference DAG or ensemble
- Benchmark p50/p95/p99 latency versus throughput with Perf Analyzer and Locust to tune queues and timeouts
- Roll out new models safely on live traffic using shadow (dark launch), canary and A/B routing patterns
Curriculum
Inference Patterns and Protocol Selection
- Trade-offs between synchronous (REST, gRPC), async queues, streaming (Kafka) and offline batch scoring
- Overcoming serialisation bottlenecks for large tensors (images/embeddings) with gRPC/Protobuf and shared memory
High-Throughput Serving with NVIDIA Triton Inference Server
- Model repository layout, ONNX/TensorRT/PyTorch/Python backends and tuning config.pbtxt
- Dynamic batching, preferred batch sizes, max queue delay and concurrent model execution on a single GPU
Python-Native Inference Graphs with BentoML, Ray Serve and vLLM
- Independently scaling CPU-bound preprocessing and GPU-bound model steps with BentoML and Ray Serve
- Continuous batching and PagedAttention with vLLM for LLM and VLM workloads
Load Testing, Capacity Planning and Caching
- Mapping p99 latency versus throughput saturation curves with Triton Perf Analyzer / Model Analyzer and Locust
- Feature caching, semantic response caching (Redis) and mitigating model cold-start times
Traffic Routing and Safe Rollout Patterns
- Shadow deployments (traffic mirroring), canary releases and weighted A/B routing
- Circuit breakers, fallback models/heuristics and graceful degradation under overload
Other Courses in This Category
-
Validating data, models and code in GitHub Actions/GitLab CI, automated PR evaluation reports, Continuous Training (CT) triggers with Airflow/Argo/Prefect and GitOps deployment, for MLOps teams.
- Live Course
- 5 modules
-
Optimised Docker images for CUDA/PyTorch, Kubernetes NVIDIA GPU Operator, MIG/time-slicing, serverless/autoscaling inference with KServe and distributed training with Kubeflow/Volcano, for platform and MLOps teams.
- Live Course
- 5 modules
-
GPU memory/compute architecture, profiling with Nsight Systems, TensorRT, FlashAttention, KV cache management, AWQ/GPTQ/FP8 quantization and speculative decoding, for engineers cutting deep learning and LLM inference costs.
- Live Course
- 5 modules
-
SageMaker Studio, Processing and Training Jobs, SageMaker Pipelines, Model Registry, Real-Time/Serverless/Async Inference, Model Monitor and IAM/VPC security, for teams running end-to-end ML on AWS.
- Live Course
- 5 modules
-
System and model telemetry with OpenTelemetry, Prometheus and Grafana, prediction logging, delayed ground-truth joining, live performance tracking and incident response, for MLOps and SRE teams.
- Live Course
- 5 modules
-
The maths of covariate, label and concept drift; PSI, KS, Wasserstein and MMD tests; label-free performance estimation with Evidently/NannyML and retraining strategies, for data scientists and MLOps teams.
- Live Course
- 5 modules
Get in touch
