Course Objective & Scope
Moving TensorFlow Serving from a single container to enterprise Kubernetes clusters introduces operational hurdles: pulling large model artefacts from S3/GCS, sharing GPU nodes, load-balancing gRPC streams and preventing pods from receiving traffic during cold starts. This course teaches cloud-native packaging and orchestration for TensorFlow services.
Participants build custom Docker images, wire up initContainers and object storage, configure HTTP/gRPC probes and L7 gRPC load balancing with Envoy/Istio, and set up autoscaling via HPA and KServe. Teams leave with a reusable Helm chart/manifest template and an operational readiness checklist.
Who Should Attend?
- MLOps and platform engineers running model services on Kubernetes
- DevOps and SRE teams onboarding AI workloads onto container infrastructure
- ML engineers standardising production deployment pipelines
- Prerequisite: basic knowledge of Docker, Kubernetes (Deployments, Services) and TensorFlow Serving
Key Takeaways
- Decouple large model weights from container images using S3/GCS or PVCs
- Tune Kubernetes startup, readiness and liveness probes around model load times
- Solve HTTP/2 gRPC connection sticking using L7 load balancing
- Schedule GPU workloads with NVIDIA GPU Operator, resource limits and node affinity
- Configure custom-metric autoscaling and canary rollouts with HPA/KEDA and KServe
Curriculum
Production Docker Packaging and Model Storage
- Selecting and customising CPU (AVX/OneDNN) and GPU TensorFlow Serving images
- Baked-in images versus mounting models from S3/GCS/MinIO and PVCs
Kubernetes Deployments, Probes and Configuration Management
- Injecting
models.configandbatching.configvia ConfigMaps - Tuning
startupProbe,readinessProbeand graceful shutdown for large models
Networking, gRPC Load Balancing and GPU Scheduling
- HTTP/2 gRPC load-balancing pitfalls in Kubernetes Services and Envoy/Ingress solutions
- NVIDIA device plugin, GPU resource limits and taints/tolerations
Autoscaling with HPA, KEDA and Custom Metrics
- Limits of CPU scaling; HPA/KEDA rules driven by QPS and queue latency
- Zero-downtime strategies for rolling updates and blue/green switches
Declarative Serving with KServe and CI/CD Templates
- Serving TensorFlow models declaratively with KServe
InferenceService - Helm chart packaging and an internal deployment checklist
Other Courses in This Category
-
Deploying PyTorch models with TorchServe: writing custom Handlers, packaging .mar files with torch-model-archiver, using the Inference and Management APIs, and managing workers and versions. For data scientists and ML engineers using PyTorch.
- Live Course
- 5 modules
-
Multi-model DAG workflows in TorchServe, acceleration with torch.compile/ONNX/TensorRT, multi-GPU and large-model parallel serving, custom Prometheus metrics and production security. For senior ML and MLOps engineers.
- Live Course
- 5 modules
-
Dynamic batching and high-volume bulk scoring in TorchServe: calibrating batch_size and max_batch_delay, writing vectorised Handlers, padding variable-length inputs and integrating queues. For ML and data engineers.
- Live Course
- 5 modules
-
Converting PyTorch, TensorFlow and scikit-learn models to ONNX, validating graphs, accelerating inference with ONNX Runtime and Execution Providers (CPU, CUDA, TensorRT) and quantization. For ML and deployment engineers.
- Live Course
- 5 modules
-
High-performance model serving with NVIDIA Triton Inference Server: Model Repository layout, config.pbtxt, dynamic batching, concurrent model execution, HTTP/gRPC clients and Perf Analyzer. For ML and MLOps engineers.
- Live Course
- 5 modules
-
Automated tuning with Triton Model Analyzer, Ensembles and Business Logic Scripting (BLS), stateful/sequence batching, shared memory (IPC) and LLM serving with TensorRT-LLM/vLLM backends. For senior ML engineers.
- Live Course
- 5 modules
Get in touch
