Model Online / On-Site
Duration
Modules 5 modules

Course Objective & Scope

TensorFlow Serving deployments handling high QPS or hosting dozens of models on shared hardware quickly hit GPU bottlenecks and latency spikes with default settings. This advanced programme dives into the server-side batching scheduler, XLA and TensorRT graph optimisations, multi-model/version labels and observability configuration.

Participants dynamically reload model pools without restarts via HandleReloadConfigRequest, calibrate batching parameters under load and monitor queue latencies with Prometheus metrics. The course concludes with a production-ready configuration bundle and a performance tuning checklist.

Who Should Attend?

  • Senior ML and MLOps engineers operating TensorFlow Serving in production
  • Performance engineers responsible for GPU efficiency and low-latency inference
  • Platform engineers managing multi-model serving infrastructure
  • Prerequisite: hands-on experience with SavedModel and basic TensorFlow Serving

Key Takeaways

  • Manage multiple models and canary version labels dynamically with ModelServerConfig
  • Optimise server-side batching parameters (batch size, timeout, threads) for specific workloads
  • Reduce inference latency using XLA, FP16/INT8 quantization and TF-TensorRT
  • Configure GPU memory allocation and CPU/GPU thread pools without contention
  • Diagnose production bottlenecks using Prometheus metrics and TensorFlow Profiler

Curriculum

TensorFlow Serving Internals and Lifecycle
  • Servable, AspiredVersionManager, Source and Loader components
  • Memory management, model load/unload ordering and zero-downtime transitions
Dynamic Multi-Model and Version Label Management
  • Multi-model definitions and stable/canary labels in models.config
  • Zero-downtime runtime updates via HandleReloadConfigRequest
Server-Side Batching and Thread Pool Tuning
  • Calibrating max_batch_size, batch_timeout_micros and max_enqueued_batches
  • Inter-op / intra-op parallelism and per_process_gpu_memory_fraction settings
Graph Optimisation: XLA, Quantization and TensorRT
  • XLA JIT compilation and Grappler optimisation passes
  • FP16/INT8 post-training quantization and GPU acceleration with TF-TensorRT
Observability, Profiling and Troubleshooting
  • Prometheus /monitoring/prometheus/metrics output and queueing metrics
  • Analysing in-graph operation timings with TensorFlow Profiler

Other Courses in This Category

  • Deploying TensorFlow Serving on Docker and Kubernetes

    Operating TensorFlow Serving at scale on Docker and Kubernetes: pulling models from object storage, GPU scheduling, readiness/liveness probes, HPA/KEDA autoscaling and KServe integration. For MLOps and platform engineers.

    Model Serving & Deployment Corporate
    Live Course
    5 modules
    View
  • TorchServe Fundamentals for PyTorch Model Serving

    Deploying PyTorch models with TorchServe: writing custom Handlers, packaging .mar files with torch-model-archiver, using the Inference and Management APIs, and managing workers and versions. For data scientists and ML engineers using PyTorch.

    Model Serving & Deployment Corporate
    Live Course
    5 modules
    View
  • Advanced TorchServe: Workflows, Optimization and Scaling

    Multi-model DAG workflows in TorchServe, acceleration with torch.compile/ONNX/TensorRT, multi-GPU and large-model parallel serving, custom Prometheus metrics and production security. For senior ML and MLOps engineers.

    Model Serving & Deployment Corporate
    Live Course
    5 modules
    View
  • Dynamic and Offline Batch Inference with TorchServe

    Dynamic batching and high-volume bulk scoring in TorchServe: calibrating batch_size and max_batch_delay, writing vectorised Handlers, padding variable-length inputs and integrating queues. For ML and data engineers.

    Model Serving & Deployment Corporate
    Live Course
    5 modules
    View
  • ONNX and ONNX Runtime Fundamentals

    Converting PyTorch, TensorFlow and scikit-learn models to ONNX, validating graphs, accelerating inference with ONNX Runtime and Execution Providers (CPU, CUDA, TensorRT) and quantization. For ML and deployment engineers.

    Model Serving & Deployment Corporate
    Live Course
    5 modules
    View
  • NVIDIA Triton Inference Server Fundamentals

    High-performance model serving with NVIDIA Triton Inference Server: Model Repository layout, config.pbtxt, dynamic batching, concurrent model execution, HTTP/gRPC clients and Perf Analyzer. For ML and MLOps engineers.

    Model Serving & Deployment Corporate
    Live Course
    5 modules
    View

Get in touch