Model Online / On-Site
Duration
Modules 5 modules

Course Objective & Scope

In LLM inference, pushing for higher throughput often degrades single-user latency; effective optimisation starts by diagnosing whether your workload is prefill-heavy (compute-bound) or decode-heavy (memory-bandwidth-bound). This course covers the roofline model, kernel-level accelerators (FlashAttention, FlashInfer, CUDA Graphs), speculative decoding techniques (draft models, EAGLE, n-gram) and choosing between tensor, pipeline and data parallelism.

Participants run benchmarks against realistic traffic profiles and find the configuration that delivers the highest tokens/s per GPU without breaching P95 TTFT and TPOT SLO targets.

Who Should Attend?

  • Senior AI and platform engineers responsible for LLM serving performance
  • ML engineers building real-time voice, chat and coding assistant infrastructure
  • MLOps teams aiming to cut GPU cost while meeting strict SLA/SLO targets
  • Prerequisite: hands-on vLLM/SGLang/TensorRT-LLM experience and basic GPU architecture knowledge

Key Takeaways

  • Diagnose prefill and decode bottlenecks using the GPU roofline model
  • Reduce decode latency with speculative decoding (EAGLE, Medusa, n-gram)
  • Choose the optimal mix of tensor parallelism, pipeline parallelism and data replicas
  • Cut CPU and kernel launch overhead with CUDA Graphs, torch.compile and attention backends
  • Plot the throughput-latency Pareto frontier under P95 TTFT and TPOT SLO constraints

Curriculum

LLM Inference Metrics and the Roofline Model
  • Defining TTFT, ITL/TPOT, E2E latency and goodput (SLO-compliant throughput)
  • FLOP/s versus HBM memory bandwidth: arithmetic intensity analysis
Kernel and Runtime Acceleration
  • FlashAttention-2/3, FlashInfer and fused MLP/norm kernels
  • CUDA Graphs, async output processing and CPU scheduling bottlenecks
Speculative Decoding Techniques
  • Draft-target models, EAGLE, Medusa and prompt lookup (n-gram) methods
  • Acceptance rates and why speculative decoding trades off at high batch sizes
Parallelism Strategies and Multi-GPU Trade-Offs
  • Communication overhead of tensor parallelism (TP) and NVLink versus PCIe impact
  • When to run independent data-parallel replicas instead of increasing TP
SLO-Driven Benchmarking and Tuning Report
  • Testing across QPS levels and prompt distributions with GenAI-Perf and vllm bench serve
  • Maximising goodput and producing an organisation-specific tuning playbook

Other Courses in This Category

  • Running AI and LLM Workloads on Kubernetes

    Operating GPU and LLM workloads on Kubernetes: NVIDIA GPU Operator, GPU sharing with MIG and time-slicing, queue management with Kueue/Volcano, fast model weight caching, GPU autoscaling and DCGM observability.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • Deploying Small Language Models on Edge and On-Device Hardware

    Running Small Language Models (SLMs) on edge devices, field hardware and NPUs/embedded GPUs: Llama 3.2, Qwen, Gemma and Phi (1B–4B), ONNX Runtime GenAI, ExecuTorch, TensorRT, Jetson and offline field scenarios.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • LLM Unit Economics: Cost per Request and Token Optimization

    Unit cost analysis and optimisation in LLM systems: self-hosted GPU vs commercial API break-even math, prompt and output token reduction, batch APIs, model cascading and AI FinOps metrics.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • LLM Model Serving and Deployment Fundamentals

    Shows how an open-weight LLM is served as an OpenAI-compatible API on a GPU server: inference engines, VRAM math, core serving settings and a first benchmark. An entry-level programme for developer and platform teams.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • LLM Serving with vLLM: Production Inference Deployment

    High-throughput LLM serving with vLLM: PagedAttention, continuous batching, tensor parallelism, prefix caching, multi-LoRA serving and Prometheus metrics. Teams leave with a production-ready vLLM deployment and benchmark report.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • Hugging Face Text Generation Inference (TGI) for LLM Serving

    Serving LLMs with Hugging Face Text Generation Inference (TGI): Docker setup, sharding, quantization, Messages API, structured output with guidance and Inference Endpoints. A hands-on programme for teams in the Hugging Face ecosystem.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View

Get in touch