Model Online / On-Site
Duration
Modules 5 modules

Course Objective & Scope

Unlocking full performance from Triton in complex production scenarios requires zero-copy tensor passing across models, conditional business logic, stateful sequence scheduling and in-flight batching for LLMs. This advanced programme covers Triton's high-end architectural capabilities through hands-on labs.

Participants build multi-stage GPU pipelines with Ensembles and Business Logic Scripting (BLS), bypass network overhead using System/CUDA Shared Memory, automate optimal batch/instance discovery with Model Analyzer and run streaming LLM services on TensorRT-LLM and vLLM backends. They leave with an advanced Triton architecture and a tuning report for their own workload.

Who Should Attend?

  • Senior ML engineers targeting high throughput and strict latency SLAs on Triton
  • AI architects building multi-stage vision, speech or LLM inference pipelines
  • MLOps and platform engineers optimising GPU cluster cost and capacity
  • Prerequisite: hands-on experience with Triton Fundamentals (Model Repository, config.pbtxt, client APIs)

Key Takeaways

  • Design multi-model pipelines with loops and conditionals using Ensemble scheduler and BLS
  • Find optimal config.pbtxt settings under latency and VRAM constraints with Model Analyzer
  • Configure the sequence batcher, response cache and priority queues
  • Eliminate large tensor transfer overhead using System and CUDA Shared Memory
  • Serve LLMs in decoupled streaming mode using TensorRT-LLM and vLLM backends

Curriculum

Ensemble Models and Business Logic Scripting (BLS)
  • Chaining pre-processing, model and post-processing steps with the Ensemble scheduler
  • Conditional and looped model execution via in-process InferenceRequest (BLS)
  • Zero-copy GPU tensor sharing across steps using DLPack
Advanced Scheduling: Sequence Batcher, Priorities and Cache
  • sequence_batching for stateful models (Direct and Oldest strategies, CORRID)
  • Multi-level priority queues, timeouts and Triton Response Cache configuration
Memory, IPC and Multi-GPU Optimisation
  • System and CUDA Shared Memory (IPC) for high-bandwidth image/audio payloads
  • NVIDIA MIG partitioning, rate limiters and multi-GPU workload placement
Automated Capacity and Config Tuning with Model Analyzer
  • Automated parameter sweeps under latency budgets and GPU VRAM limits
  • Interpreting Pareto frontier reports to pick the optimal production config
LLM and Generative AI Serving (TensorRT-LLM & vLLM)
  • Decoupled transaction policy and gRPC streaming token generation
  • Serving LLMs with in-flight/continuous batching, paged KV cache and tensor parallelism

Other Courses in This Category

  • Integrating Multi-Framework Models with NVIDIA Triton

    Unifying PyTorch, TensorFlow, ONNX, TensorRT, OpenVINO and scikit-learn/XGBoost (FIL) models alongside custom Python backends on a single Triton server. For ML and MLOps engineers managing heterogeneous model portfolios.

    Model Serving & Deployment Corporate
    Live Course
    5 modules
    View
  • Serving Machine Learning and LLM Models with FastAPI

    Building production-ready ML and LLM inference services with FastAPI: model loading via lifespan, Pydantic v2 schemas, async/threadpool execution, SSE streaming, Gunicorn/Uvicorn tuning and Prometheus metrics. For developers and ML engineers.

    Model Serving & Deployment Corporate
    Live Course
    5 modules
    View
  • Serving Machine Learning Models with Flask

    Building lightweight ML prediction services with Flask and WSGI: Application Factory, Blueprints, Marshmallow/Pydantic validation, Gunicorn worker/thread tuning and Docker packaging. For data scientists and Python developers.

    Model Serving & Deployment Corporate
    Live Course
    5 modules
    View
  • Distributed Model and LLM Serving with Ray Serve

    Python-native distributed model and LLM serving with Ray Serve: DeploymentGraph composition, fractional GPU sharing, dynamic batching, replica autoscaling, vLLM integration and KubeRay. For ML and platform engineers.

    Model Serving & Deployment Corporate
    Live Course
    5 modules
    View
  • Model Versioning, Rollout Strategies and Safe Replacement

    Updating production models safely without downtime: MLflow Model Registry, semantic versioning, shadow deployments, canary rollouts, blue/green switches, A/B testing and automated rollback. For MLOps and ML engineers.

    Model Serving & Deployment Corporate
    Live Course
    5 modules
    View
  • Scaling Deployed Machine Learning and LLM Services

    Horizontal and vertical scaling for live model services: autoscaling on queue/GPU metrics with Kubernetes HPA, KEDA and Knative, reducing cold starts, MIG/time-slicing and load balancing. For MLOps, SRE and platform engineers.

    Model Serving & Deployment Corporate
    Live Course
    5 modules
    View

Get in touch