Model Online / On-Site
Duration
Modules 5 modules

Course Objective & Scope

Real-world AI systems rarely consist of a single isolated model; they require multi-step pipelines chaining OCR to text classification, large models spanning multiple GPUs and strict latency targets. This advanced course focuses on adapting TorchServe to complex production topologies and extracting maximum hardware efficiency.

Participants build DAG-based ensembles and pipelines with TorchServe Workflows, accelerate inference using torch.compile, Torch-TensorRT and IPEX, distribute large models across GPUs with torchrun and device_ids, and configure Prometheus metrics alongside SSL/token security. They leave with a reference architecture and a production hardening checklist.

Who Should Attend?

  • Senior ML and AI engineers serving PyTorch models at high scale
  • MLOps and platform engineers operating TorchServe on GPU clusters
  • Technical leads designing multi-stage computer vision or NLP inference pipelines
  • Prerequisite: TorchServe fundamentals (Handlers and .mar archives) and PyTorch experience

Key Takeaways

  • Build multi-model DAG pipelines using TorchServe Workflows and .war archives
  • Accelerate inference by integrating torch.compile, ONNX Runtime and TensorRT
  • Serve large models with multi-GPU pinning and tensor/pipeline parallelism
  • Export custom business and model metrics in Prometheus format
  • Harden enterprise deployments with SSL/TLS, token authentication and KServe integration

Curriculum

Multi-Model Orchestration with TorchServe Workflows
  • Defining DAGs in YAML: sequential, parallel and ensemble model flows
  • Packaging .war files with torch-workflow-archiver and managing the Workflow API
Compiler and Hardware Acceleration Techniques
  • Using torch.compile, Torch-TensorRT and ONNX Runtime inside Handlers
  • FP16/BF16 autocast, INT8 quantization and Intel Extension for PyTorch (IPEX) on CPU
Multi-GPU Management and Large Model Serving
  • Per-worker GPU pinning (device_ids) and memory isolation
  • Distributed inference with torchrun and Accelerate/DeepSpeed integration
  • Streaming responses for generative model serving
Advanced Observability, Profiling and Debugging
  • Defining custom counter, gauge and histogram metrics via the Metrics API
  • Inspecting worker bottlenecks with PyTorch Profiler and TorchServe logs
Production Security and Kubernetes/KServe Deployment
  • SSL/TLS certificates, key/token authentication and locking down the Management API
  • Scalable TorchServe topologies on Kubernetes and KServe

Other Courses in This Category

  • Dynamic and Offline Batch Inference with TorchServe

    Dynamic batching and high-volume bulk scoring in TorchServe: calibrating batch_size and max_batch_delay, writing vectorised Handlers, padding variable-length inputs and integrating queues. For ML and data engineers.

    Model Serving & Deployment Corporate
    Live Course
    5 modules
    View
  • ONNX and ONNX Runtime Fundamentals

    Converting PyTorch, TensorFlow and scikit-learn models to ONNX, validating graphs, accelerating inference with ONNX Runtime and Execution Providers (CPU, CUDA, TensorRT) and quantization. For ML and deployment engineers.

    Model Serving & Deployment Corporate
    Live Course
    5 modules
    View
  • NVIDIA Triton Inference Server Fundamentals

    High-performance model serving with NVIDIA Triton Inference Server: Model Repository layout, config.pbtxt, dynamic batching, concurrent model execution, HTTP/gRPC clients and Perf Analyzer. For ML and MLOps engineers.

    Model Serving & Deployment Corporate
    Live Course
    5 modules
    View
  • Advanced NVIDIA Triton: Ensembles, BLS, LLMs and Tuning

    Automated tuning with Triton Model Analyzer, Ensembles and Business Logic Scripting (BLS), stateful/sequence batching, shared memory (IPC) and LLM serving with TensorRT-LLM/vLLM backends. For senior ML engineers.

    Model Serving & Deployment Corporate
    Live Course
    5 modules
    View
  • Integrating Multi-Framework Models with NVIDIA Triton

    Unifying PyTorch, TensorFlow, ONNX, TensorRT, OpenVINO and scikit-learn/XGBoost (FIL) models alongside custom Python backends on a single Triton server. For ML and MLOps engineers managing heterogeneous model portfolios.

    Model Serving & Deployment Corporate
    Live Course
    5 modules
    View
  • Serving Machine Learning and LLM Models with FastAPI

    Building production-ready ML and LLM inference services with FastAPI: model loading via lifespan, Pydantic v2 schemas, async/threadpool execution, SSE streaming, Gunicorn/Uvicorn tuning and Prometheus metrics. For developers and ML engineers.

    Model Serving & Deployment Corporate
    Live Course
    5 modules
    View

Get in touch