Model Online / On-Site
Duration
Modules 5 modules

Course Objective & Scope

Simply increasing the batch_size parameter in TorchServe yields no speedup—and often breaks requests—if the Handler code was written for single inputs. This course covers both online dynamic batching that multiplies GPU efficiency and offline/micro-batch scoring architectures capable of processing millions of records.

Participants write vectorised preprocess and postprocess methods that merge request lists into a single tensor, apply dynamic padding for variable-length text and images, handle partial batch failures cleanly and connect TorchServe to Airflow/Kafka pipelines. They leave with a load-tested batch handler template.

Who Should Attend?

  • ML engineers looking to boost GPU utilisation and throughput in TorchServe
  • Data engineers running bulk scoring over large document, image or tabular datasets
  • MLOps specialists aiming to reduce inference infrastructure costs
  • Prerequisite: basic PyTorch and TorchServe Handler knowledge

Key Takeaways

  • Configure the dynamic batching flow between the TorchServe frontend queue and backend workers
  • Write vectorised preprocess, inference and postprocess methods for multi-request batches
  • Tensorise variable-length NLP inputs and multi-resolution images safely with dynamic padding
  • Calibrate batch_size and max_batch_delay parameters against latency SLAs
  • Build asynchronous queue and micro-batch scoring pipelines for large datasets

Curriculum

How TorchServe's Batching Mechanism Works
  • Frontend job queue, batch_size, max_batch_delay and worker communication
  • Architectural differences between online dynamic batching and offline bulk scoring
Developing Batch-Aware Vectorised Handlers
  • Parsing the data list and combining tensors with torch.stack / torch.cat
  • Mapping prediction outputs back to individual requests while preserving order
  • Isolating malformed inputs (partial failures) inside a batch
Variable-Length Inputs: NLP and Image Padding
  • Dynamic padding, attention masks and length bucketing in tokenizers
  • Resize/pad strategies for variable-resolution images and avoiding CPU bottlenecks
Parameter Calibration and Memory Management
  • Preventing CUDA Out-of-Memory (OOM) errors and using torch.inference_mode
  • Benchmarking batch_size and max_batch_delay combinations with Locust/wrk
High-Volume Offline and Micro-Batch Pipelines
  • Reading from S3/Parquet and dispatching via async HTTP/gRPC client pools
  • Bulk scoring architectures driving TorchServe from Airflow or Kafka consumers

Other Courses in This Category

  • ONNX and ONNX Runtime Fundamentals

    Converting PyTorch, TensorFlow and scikit-learn models to ONNX, validating graphs, accelerating inference with ONNX Runtime and Execution Providers (CPU, CUDA, TensorRT) and quantization. For ML and deployment engineers.

    Model Serving & Deployment Corporate
    Live Course
    5 modules
    View
  • NVIDIA Triton Inference Server Fundamentals

    High-performance model serving with NVIDIA Triton Inference Server: Model Repository layout, config.pbtxt, dynamic batching, concurrent model execution, HTTP/gRPC clients and Perf Analyzer. For ML and MLOps engineers.

    Model Serving & Deployment Corporate
    Live Course
    5 modules
    View
  • Advanced NVIDIA Triton: Ensembles, BLS, LLMs and Tuning

    Automated tuning with Triton Model Analyzer, Ensembles and Business Logic Scripting (BLS), stateful/sequence batching, shared memory (IPC) and LLM serving with TensorRT-LLM/vLLM backends. For senior ML engineers.

    Model Serving & Deployment Corporate
    Live Course
    5 modules
    View
  • Integrating Multi-Framework Models with NVIDIA Triton

    Unifying PyTorch, TensorFlow, ONNX, TensorRT, OpenVINO and scikit-learn/XGBoost (FIL) models alongside custom Python backends on a single Triton server. For ML and MLOps engineers managing heterogeneous model portfolios.

    Model Serving & Deployment Corporate
    Live Course
    5 modules
    View
  • Serving Machine Learning and LLM Models with FastAPI

    Building production-ready ML and LLM inference services with FastAPI: model loading via lifespan, Pydantic v2 schemas, async/threadpool execution, SSE streaming, Gunicorn/Uvicorn tuning and Prometheus metrics. For developers and ML engineers.

    Model Serving & Deployment Corporate
    Live Course
    5 modules
    View
  • Serving Machine Learning Models with Flask

    Building lightweight ML prediction services with Flask and WSGI: Application Factory, Blueprints, Marshmallow/Pydantic validation, Gunicorn worker/thread tuning and Docker packaging. For data scientists and Python developers.

    Model Serving & Deployment Corporate
    Live Course
    5 modules
    View

Get in touch