Course Objective & Scope
TensorFlow Serving deployments handling high QPS or hosting dozens of models on shared hardware quickly hit GPU bottlenecks and latency spikes with default settings. This advanced programme dives into the server-side batching scheduler, XLA and TensorRT graph optimisations, multi-model/version labels and observability configuration.
Participants dynamically reload model pools without restarts via HandleReloadConfigRequest, calibrate batching parameters under load and monitor queue latencies with Prometheus metrics. The course concludes with a production-ready configuration bundle and a performance tuning checklist.
Who Should Attend?
- Senior ML and MLOps engineers operating TensorFlow Serving in production
- Performance engineers responsible for GPU efficiency and low-latency inference
- Platform engineers managing multi-model serving infrastructure
- Prerequisite: hands-on experience with SavedModel and basic TensorFlow Serving
Key Takeaways
- Manage multiple models and canary version labels dynamically with
ModelServerConfig - Optimise server-side batching parameters (batch size, timeout, threads) for specific workloads
- Reduce inference latency using XLA, FP16/INT8 quantization and TF-TensorRT
- Configure GPU memory allocation and CPU/GPU thread pools without contention
- Diagnose production bottlenecks using Prometheus metrics and TensorFlow Profiler
Curriculum
TensorFlow Serving Internals and Lifecycle
- Servable, AspiredVersionManager, Source and Loader components
- Memory management, model load/unload ordering and zero-downtime transitions
Dynamic Multi-Model and Version Label Management
- Multi-model definitions and
stable/canarylabels inmodels.config - Zero-downtime runtime updates via
HandleReloadConfigRequest
Server-Side Batching and Thread Pool Tuning
- Calibrating
max_batch_size,batch_timeout_microsandmax_enqueued_batches - Inter-op / intra-op parallelism and
per_process_gpu_memory_fractionsettings
Graph Optimisation: XLA, Quantization and TensorRT
- XLA JIT compilation and Grappler optimisation passes
- FP16/INT8 post-training quantization and GPU acceleration with TF-TensorRT
Observability, Profiling and Troubleshooting
- Prometheus
/monitoring/prometheus/metricsoutput and queueing metrics - Analysing in-graph operation timings with TensorFlow Profiler
Other Courses in This Category
-
Operating TensorFlow Serving at scale on Docker and Kubernetes: pulling models from object storage, GPU scheduling, readiness/liveness probes, HPA/KEDA autoscaling and KServe integration. For MLOps and platform engineers.
- Live Course
- 5 modules
-
Deploying PyTorch models with TorchServe: writing custom Handlers, packaging .mar files with torch-model-archiver, using the Inference and Management APIs, and managing workers and versions. For data scientists and ML engineers using PyTorch.
- Live Course
- 5 modules
-
Multi-model DAG workflows in TorchServe, acceleration with torch.compile/ONNX/TensorRT, multi-GPU and large-model parallel serving, custom Prometheus metrics and production security. For senior ML and MLOps engineers.
- Live Course
- 5 modules
-
Dynamic batching and high-volume bulk scoring in TorchServe: calibrating batch_size and max_batch_delay, writing vectorised Handlers, padding variable-length inputs and integrating queues. For ML and data engineers.
- Live Course
- 5 modules
-
Converting PyTorch, TensorFlow and scikit-learn models to ONNX, validating graphs, accelerating inference with ONNX Runtime and Execution Providers (CPU, CUDA, TensorRT) and quantization. For ML and deployment engineers.
- Live Course
- 5 modules
-
High-performance model serving with NVIDIA Triton Inference Server: Model Repository layout, config.pbtxt, dynamic batching, concurrent model execution, HTTP/gRPC clients and Perf Analyzer. For ML and MLOps engineers.
- Live Course
- 5 modules
Get in touch
