Model Online / On-Site
Duration
Modules 5 modules

Course Objective & Scope

When an organisation needs to serve PyTorch, ONNX and TensorRT models side by side on shared GPUs with high utilisation, NVIDIA Triton Inference Server is the industry standard. This course teaches Triton's architecture, Model Repository structure and core performance features hands-on from the ground up.

Participants author config.pbtxt files, enable dynamic batching and concurrent execution via instance_group, query models over HTTP and gRPC with tritonclient and run baseline benchmarks with Perf Analyzer. Teams leave with a ready-to-use Triton repository template built around their own models.

Who Should Attend?

  • ML engineers looking to serve models efficiently on GPUs and CPUs
  • MLOps and platform engineers building a unified multi-framework serving layer
  • Developers moving from simple Python web servers to enterprise inference runtimes
  • Prerequisite: Python, Docker and basic model deployment knowledge

Key Takeaways

  • Structure a standard Triton Model Repository and configure version policies
  • Define input/output tensors, data types and shapes in config.pbtxt
  • Increase GPU utilisation using dynamic_batching and instance_group settings
  • Send synchronous and asynchronous inference requests with Python tritonclient (HTTP and gRPC)
  • Profile baseline latency and throughput using Perf Analyzer and Prometheus metrics

Curriculum

Triton Architecture and Model Repository Layout
  • Supported backends, request schedulers and memory architecture in Triton
  • Model directory hierarchy, version folders and launching the NGC container
Model Configuration with config.pbtxt
  • Defining platform/backend, max_batch_size, input and output
  • Auto-generated configs and dynamic shape (-1) settings
Dynamic Batching and Concurrent Model Execution
  • The dynamic_batching block, preferred batch sizes and queue delay
  • Running multiple model instances on a single GPU/CPU with instance_group
HTTP/REST (KServe v2) and gRPC Client Development
  • Open Inference Protocol (v2) health, metadata and infer endpoints
  • Building Python clients with tritonclient.http and tritonclient.grpc
Model Control Modes, Metrics and Perf Analyzer
  • Loading and unloading models at runtime using NONE, POLL and EXPLICIT modes
  • Prometheus metrics and baseline load testing with Perf Analyzer

Other Courses in This Category

  • Advanced NVIDIA Triton: Ensembles, BLS, LLMs and Tuning

    Automated tuning with Triton Model Analyzer, Ensembles and Business Logic Scripting (BLS), stateful/sequence batching, shared memory (IPC) and LLM serving with TensorRT-LLM/vLLM backends. For senior ML engineers.

    Model Serving & Deployment Corporate
    Live Course
    5 modules
    View
  • Integrating Multi-Framework Models with NVIDIA Triton

    Unifying PyTorch, TensorFlow, ONNX, TensorRT, OpenVINO and scikit-learn/XGBoost (FIL) models alongside custom Python backends on a single Triton server. For ML and MLOps engineers managing heterogeneous model portfolios.

    Model Serving & Deployment Corporate
    Live Course
    5 modules
    View
  • Serving Machine Learning and LLM Models with FastAPI

    Building production-ready ML and LLM inference services with FastAPI: model loading via lifespan, Pydantic v2 schemas, async/threadpool execution, SSE streaming, Gunicorn/Uvicorn tuning and Prometheus metrics. For developers and ML engineers.

    Model Serving & Deployment Corporate
    Live Course
    5 modules
    View
  • Serving Machine Learning Models with Flask

    Building lightweight ML prediction services with Flask and WSGI: Application Factory, Blueprints, Marshmallow/Pydantic validation, Gunicorn worker/thread tuning and Docker packaging. For data scientists and Python developers.

    Model Serving & Deployment Corporate
    Live Course
    5 modules
    View
  • Distributed Model and LLM Serving with Ray Serve

    Python-native distributed model and LLM serving with Ray Serve: DeploymentGraph composition, fractional GPU sharing, dynamic batching, replica autoscaling, vLLM integration and KubeRay. For ML and platform engineers.

    Model Serving & Deployment Corporate
    Live Course
    5 modules
    View
  • Model Versioning, Rollout Strategies and Safe Replacement

    Updating production models safely without downtime: MLflow Model Registry, semantic versioning, shadow deployments, canary rollouts, blue/green switches, A/B testing and automated rollback. For MLOps and ML engineers.

    Model Serving & Deployment Corporate
    Live Course
    5 modules
    View

Get in touch