Course Objective & Scope
A highly accurate model becomes unsustainable in production when its response time breaches SLA targets or multiplies the GPU bill. This course examines the engineering trade-offs between p50/p95/p99 latency, request/token throughput per second and cost per 1,000 predictions through analytical models and hands-on benchmarks.
Participants run load tests with Locust and Perf Analyzer to pinpoint bottlenecks, then experiment with dynamic/continuous batching, INT8/FP8 quantization, KV and semantic caching, and CPU/GPU instance selection. Working on a case adapted to the organisation, teams leave with a reusable capacity-cost calculator and an optimisation checklist.
Who Should Attend?
- ML and MLOps engineers responsible for inference performance and cost
- Platform and infrastructure engineers managing cloud and GPU resources
- Technical leads and solution architects of high-traffic AI products
- Prerequisite: Python, container fundamentals and basic model serving experience
Key Takeaways
- Measure p50/p95/p99 latency, TTFT/ITL, throughput and unit cost accurately
- Distinguish compute-bound and memory-bound bottlenecks through load testing
- Increase GPU utilisation with dynamic batching and concurrency tuning
- Evaluate the accuracy-versus-speed impact of quantization, distillation and caching
- Make cost-effective choices across CPU, GPU, spot and serverless options based on traffic profiles
Curriculum
Fundamentals of Performance and Cost Metrics
- Tail latency (p95, p99), QPS, TTFT/ITL and GPU utilisation concepts
- Unit cost formula: hourly hardware rate, utilisation ratio and idle cost
- Analysing compute-bound versus memory-bandwidth-bound workloads
Benchmarking, Profiling and Load Testing
- Realistic traffic simulation with Locust, k6 and Triton Perf Analyzer
- Pinpointing bottlenecks using PyTorch Profiler and system metrics
- Finding the saturation point and queueing effects under SLA constraints
Batching and Concurrency Optimisation
- Static, dynamic and continuous batching mechanisms for LLMs
- Impact of batch size and max queue delay on the latency-throughput curve
- Async I/O, pre/post-processing parallelism and multi-instance serving
Model Compression and Caching Strategies
- FP16/BF16, INT8/INT4 quantization and ONNX Runtime / TensorRT acceleration
- Exact-match, embedding-based semantic cache and prompt/KV caching
- Model routing to smaller models and speculative decoding
Hardware Selection and Capacity Planning Workshop
- Comparing CPU vs GPU, GPU tiers and MIG partitioning
- On-demand, reserved, spot and scale-to-zero serverless cost models
- SLA-versus-cost simulation and checklist creation on an organisational case
Other Courses in This Category
-
End-to-end security for inference services: mTLS, OAuth2/JWT, rate limiting, safe serialization, container hardening, defence against model extraction/adversarial attacks and data privacy. For ML, platform and AppSec engineers.
- Live Course
- 5 modules
-
Exporting TensorFlow and Keras models as SavedModel, defining SignatureDefs and serving them over REST and gRPC with TensorFlow Serving. For data scientists and ML engineers starting to deploy TensorFlow models.
- Live Course
- 5 modules
-
Multi-model configurations, dynamic batching, XLA/TensorRT and mixed-precision acceleration, GPU memory tuning, Prometheus metrics and custom servables in TensorFlow Serving. For engineers operating high-traffic TF services.
- Live Course
- 5 modules
-
Operating TensorFlow Serving at scale on Docker and Kubernetes: pulling models from object storage, GPU scheduling, readiness/liveness probes, HPA/KEDA autoscaling and KServe integration. For MLOps and platform engineers.
- Live Course
- 5 modules
-
Deploying PyTorch models with TorchServe: writing custom Handlers, packaging .mar files with torch-model-archiver, using the Inference and Management APIs, and managing workers and versions. For data scientists and ML engineers using PyTorch.
- Live Course
- 5 modules
-
Multi-model DAG workflows in TorchServe, acceleration with torch.compile/ONNX/TensorRT, multi-GPU and large-model parallel serving, custom Prometheus metrics and production security. For senior ML and MLOps engineers.
- Live Course
- 5 modules
Get in touch
