Model Online / On-Site
Duration
Modules 5 modules

Course Objective & Scope

Kubernetes clusters built for stateless microservices struggle with AI workloads due to multi-gigabyte container images, massive model weights, scarce GPU resources and gang-scheduling needs. This course teaches how to turn a new or existing Kubernetes cluster into a production-ready GPU platform for both real-time LLM inference and batch/fine-tuning jobs.

In hands-on labs participants configure the NVIDIA GPU Operator, partition GPUs with MIG and time-slicing, accelerate model loading via PVCs, manage team GPU quotas with Kueue, run multi-node inference with LeaderWorkerSet and collect GPU telemetry with DCGM Exporter and Prometheus.

Who Should Attend?

  • Platform, DevOps and SRE engineers managing Kubernetes clusters
  • MLOps and LLMOps teams serving and training models on Kubernetes
  • Infrastructure architects sharing an internal GPU pool across multiple teams
  • Prerequisite: familiarity with core Kubernetes objects (Pod, Deployment, Service, PVC) and Helm

Key Takeaways

  • Manage drivers, container runtime and device plugins with the NVIDIA GPU Operator
  • Share a single GPU across smaller models safely using MIG, time-slicing and MPS
  • Cut pod cold-start times from minutes to seconds by caching model weights
  • Implement priority, quotas and batch GPU scheduling with Kueue or Volcano
  • Autoscale LLM services using DCGM metrics and custom inference queue metrics

Curriculum

GPU Management on Kubernetes and the GPU Operator
  • NVIDIA GPU Operator components, Node Feature Discovery and taints/tolerations
  • Topology-aware scheduling (Topology Manager, NUMA and NVLink)
GPU Sharing: MIG, Time-Slicing and MPS
  • Hardware-isolated Multi-Instance GPU (MIG) profiles
  • Time-slicing and MPS for embedding, reranker and development workloads
Model Weight Storage and Fast Cold-Starts
  • ReadWriteMany PVCs, local NVMe caches and OCI model artifacts
  • Image streaming and pre-pull strategies for large container images
Workload Scheduling and Multi-Node Inference
  • Multi-tenant GPU quotas, preemption and gang scheduling with Kueue and Volcano
  • Serving large LLMs across nodes with LeaderWorkerSet (LWS) and Ray
GPU Observability and Autoscaling
  • Tracking SM utilisation, VRAM, temperature and Xid hardware errors with DCGM Exporter
  • Scaling on vLLM queue depth and KV cache usage with KEDA and HPA

Other Courses in This Category

  • Deploying Small Language Models on Edge and On-Device Hardware

    Running Small Language Models (SLMs) on edge devices, field hardware and NPUs/embedded GPUs: Llama 3.2, Qwen, Gemma and Phi (1B–4B), ONNX Runtime GenAI, ExecuTorch, TensorRT, Jetson and offline field scenarios.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • LLM Unit Economics: Cost per Request and Token Optimization

    Unit cost analysis and optimisation in LLM systems: self-hosted GPU vs commercial API break-even math, prompt and output token reduction, batch APIs, model cascading and AI FinOps metrics.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • LLM Model Serving and Deployment Fundamentals

    Shows how an open-weight LLM is served as an OpenAI-compatible API on a GPU server: inference engines, VRAM math, core serving settings and a first benchmark. An entry-level programme for developer and platform teams.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • LLM Serving with vLLM: Production Inference Deployment

    High-throughput LLM serving with vLLM: PagedAttention, continuous batching, tensor parallelism, prefix caching, multi-LoRA serving and Prometheus metrics. Teams leave with a production-ready vLLM deployment and benchmark report.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • Hugging Face Text Generation Inference (TGI) for LLM Serving

    Serving LLMs with Hugging Face Text Generation Inference (TGI): Docker setup, sharding, quantization, Messages API, structured output with guidance and Inference Endpoints. A hands-on programme for teams in the Hugging Face ecosystem.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • Running Local LLMs with Ollama

    Running open-weight models on laptops, workstations or servers with Ollama: model library, Modelfile, REST/OpenAI-compatible API, Open WebUI, embeddings and a shared team setup. For fast prototyping and confidential-data scenarios.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View

Get in touch