Model Online / On-Site
Duration
Modules 5 modules

Course Objective & Scope

When PyTorch models move to production, preprocessing code, model weights and auxiliary files often end up scattered across repositories. TorchServe packages all of these into a single .mar archive and exposes ready-made Inference and Management APIs. This course walks hands-on through every step from archiving a PyTorch model to running a live service.

Participants write built-in and custom BaseHandler classes, generate model archives with torch-model-archiver and configure ports and worker counts via config.properties. They leave with a reusable handler and deployment template tailored to one of the organisation's PyTorch models.

Who Should Attend?

  • Data scientists who want to expose their PyTorch models as services
  • ML engineers standardising how PyTorch models are packaged
  • Software and platform developers containerising model APIs
  • Prerequisite: Python and basic PyTorch (nn.Module, state_dict) knowledge

Key Takeaways

  • Prepare eager-mode (state_dict) and TorchScript models for TorchServe
  • Write custom Handlers implementing initialize, preprocess, inference and postprocess
  • Bundle weights, code and extra assets into a .mar archive with torch-model-archiver
  • Query predictions and scale workers dynamically through the Inference, Management and Metrics APIs
  • Run TorchServe in Docker with essential production settings

Curriculum

TorchServe Architecture and Core Concepts
  • Java frontend, Python backend workers and the Model Store layout
  • Preparing eager-mode (state_dict) and TorchScript models
Built-In and Custom Handler Development
  • Built-in handlers such as image_classifier and text_classifier
  • Subclassing BaseHandler to implement preprocess, inference and postprocess
Model Packaging with torch-model-archiver
  • Building .mar files with --extra-files, --requirements-file and version tags
  • Inspecting archive contents and testing locally
Using the Inference and Management APIs
  • Prediction, explanations and health-check calls over REST and gRPC
  • Registering, unregistering and scaling model workers via the Management API
Configuration and Docker Deployment
  • Configuring ports, CORS, timeouts and snapshots in config.properties
  • Running the service with the official TorchServe Docker image

Other Courses in This Category

  • Advanced TorchServe: Workflows, Optimization and Scaling

    Multi-model DAG workflows in TorchServe, acceleration with torch.compile/ONNX/TensorRT, multi-GPU and large-model parallel serving, custom Prometheus metrics and production security. For senior ML and MLOps engineers.

    Model Serving & Deployment Corporate
    Live Course
    5 modules
    View
  • Dynamic and Offline Batch Inference with TorchServe

    Dynamic batching and high-volume bulk scoring in TorchServe: calibrating batch_size and max_batch_delay, writing vectorised Handlers, padding variable-length inputs and integrating queues. For ML and data engineers.

    Model Serving & Deployment Corporate
    Live Course
    5 modules
    View
  • ONNX and ONNX Runtime Fundamentals

    Converting PyTorch, TensorFlow and scikit-learn models to ONNX, validating graphs, accelerating inference with ONNX Runtime and Execution Providers (CPU, CUDA, TensorRT) and quantization. For ML and deployment engineers.

    Model Serving & Deployment Corporate
    Live Course
    5 modules
    View
  • NVIDIA Triton Inference Server Fundamentals

    High-performance model serving with NVIDIA Triton Inference Server: Model Repository layout, config.pbtxt, dynamic batching, concurrent model execution, HTTP/gRPC clients and Perf Analyzer. For ML and MLOps engineers.

    Model Serving & Deployment Corporate
    Live Course
    5 modules
    View
  • Advanced NVIDIA Triton: Ensembles, BLS, LLMs and Tuning

    Automated tuning with Triton Model Analyzer, Ensembles and Business Logic Scripting (BLS), stateful/sequence batching, shared memory (IPC) and LLM serving with TensorRT-LLM/vLLM backends. For senior ML engineers.

    Model Serving & Deployment Corporate
    Live Course
    5 modules
    View
  • Integrating Multi-Framework Models with NVIDIA Triton

    Unifying PyTorch, TensorFlow, ONNX, TensorRT, OpenVINO and scikit-learn/XGBoost (FIL) models alongside custom Python backends on a single Triton server. For ML and MLOps engineers managing heterogeneous model portfolios.

    Model Serving & Deployment Corporate
    Live Course
    5 modules
    View

Get in touch