Model Online / On-Site
Duration
Modules 5 modules

Course Objective & Scope

In high-volume enterprise AI platforms, LLMs run alongside embedding, reranker, OCR, speech and classic ML models on the same GPU fleet. This course takes a deep dive into NVIDIA Triton Inference Server's multi-backend engine (TensorRT-LLM, ONNX Runtime, PyTorch, Python/vLLM) and KServe's standardised model deployment layer on Kubernetes.

In the labs participants build dynamic batching and multi-model ensemble pipelines on Triton, tune instance counts and batch sizes with Triton Model Analyzer, and operate the services under KServe InferenceService with canary rollouts and autoscaling.

Who Should Attend?

  • MLOps and platform engineers building high-performance inference infrastructure
  • AI engineers serving multiple model types on NVIDIA GPU clusters
  • Infrastructure architects managing Kubernetes-based model serving platforms
  • Prerequisite: Docker, Kubernetes and foundational GPU/model serving knowledge

Key Takeaways

  • Configure Triton model repositories and config.pbtxt parameters
  • Run diverse models on the same server with TensorRT-LLM, ONNX and Python backends
  • Maximise GPU utilisation using dynamic batching, instance groups and Model Analyzer
  • Deploy services on KServe in both Serverless (Knative) and RawDeployment modes
  • Implement pre/post-processing transformers, canary rollouts and the OpenInference protocol

Curriculum

NVIDIA Triton Architecture and Model Repository
  • Model repository layout, versioning and config.pbtxt
  • HTTP/REST, gRPC and shared-memory protocols
Multi-Backend Serving and Ensemble Pipelines
  • ONNX Runtime, TensorRT, Python and TensorRT-LLM/vLLM backends
  • Chaining tokenizer, model and post-processing with BLS and ensembles
Batching, Concurrency and Model Analyzer
  • Dynamic batching, sequence batcher and multi-instance_group settings
  • Profiling latency and throughput with Triton Model Analyzer and Perf Analyzer
KServe Architecture and ServingRuntimes
  • InferenceService, ServingRuntime and storage initializer mechanics
  • Comparing Knative Serverless (scale-to-zero) with RawDeployment
Production Operations and Rollout Strategies
  • Canary traffic splitting, custom predictor/transformer and GPU metrics
  • Triton + KServe reference template for an enterprise serving platform

Other Courses in This Category

  • KV Cache Management and Continuous Batching Optimization for LLM Inference

    Solving memory and concurrency bottlenecks in LLM serving: KV cache memory math, PagedAttention, SGLang RadixAttention, prefix caching, FP8 KV cache quantization, continuous batching and chunked prefill tuning.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • LLM Inference Throughput and Latency Optimization

    Optimising TTFT, ITL and total throughput in LLM services: compute-bound vs memory-bandwidth-bound bottleneck analysis, FlashAttention/FlashInfer, CUDA Graphs, speculative decoding, parallelism tuning and SLO-driven load testing.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • Running AI and LLM Workloads on Kubernetes

    Operating GPU and LLM workloads on Kubernetes: NVIDIA GPU Operator, GPU sharing with MIG and time-slicing, queue management with Kueue/Volcano, fast model weight caching, GPU autoscaling and DCGM observability.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • Deploying Small Language Models on Edge and On-Device Hardware

    Running Small Language Models (SLMs) on edge devices, field hardware and NPUs/embedded GPUs: Llama 3.2, Qwen, Gemma and Phi (1B–4B), ONNX Runtime GenAI, ExecuTorch, TensorRT, Jetson and offline field scenarios.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • LLM Unit Economics: Cost per Request and Token Optimization

    Unit cost analysis and optimisation in LLM systems: self-hosted GPU vs commercial API break-even math, prompt and output token reduction, batch APIs, model cascading and AI FinOps metrics.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • LLM Model Serving and Deployment Fundamentals

    Shows how an open-weight LLM is served as an OpenAI-compatible API on a GPU server: inference engines, VRAM math, core serving settings and a first benchmark. An entry-level programme for developer and platform teams.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View

Get in touch