Model Online / On-Site
Duration
Modules 5 modules

Course Objective & Scope

In long-context, multi-user LLM applications, the primary consumer of GPU memory is often not the model weights but the KV cache that grows with every active request. This advanced programme covers KV cache math across MHA, GQA and MLA attention designs, PagedAttention and RadixAttention algorithms, and the progression from static batching to continuous batching and chunked prefill.

Participants test RAG and multi-turn agent workloads with long system prompts on vLLM and SGLang, combining prefix caching, FP8 KV cache quantization and scheduler tuning to multiply concurrent request capacity on the same GPU.

Who Should Attend?

  • AI and platform engineers optimising LLM inference engines in production
  • ML engineers building long-document RAG and multi-step agent infrastructure
  • MLOps teams looking to increase concurrent users per GPU
  • Prerequisite: Transformer architecture basics and hands-on experience with vLLM/TGI/SGLang

Key Takeaways

  • Calculate per-token KV cache memory footprint across MHA, GQA and MLA architectures
  • Eliminate memory fragmentation using PagedAttention and block-level allocation
  • Structure prompts to maximise hit rates in automatic prefix caching and RadixAttention
  • Apply FP8 KV cache quantization to expand maximum concurrency
  • Balance prefill and decode phases using continuous batching, max-num-batched-tokens and chunked prefill

Curriculum

Anatomy of the KV Cache and Attention Designs
  • Role of K and V matrices during prefill and decode phases
  • Memory comparison of MHA, MQA, GQA, sliding window and DeepSeek MLA
PagedAttention and Block Memory Management
  • Internal and external memory fragmentation and virtual block tables
  • Preemption (recompute versus swap) and KV cache occupancy metrics
Prefix Caching and RadixAttention
  • How vLLM Automatic Prefix Caching (APC) and SGLang RadixAttention work
  • Designing system prompts, few-shot examples and multi-turn history for high cache hit rates
KV Cache Compression and Offloading
  • FP8 KV cache quantization and its accuracy/capacity impact
  • CPU RAM/NVMe KV cache offloading and LMCache approaches
Continuous Batching and Chunked Prefill
  • Iteration-level scheduling, max-num-seqs and max-num-batched-tokens
  • Chunked prefill and disaggregated serving to prevent long prompts from stalling decode

Other Courses in This Category

  • LLM Inference Throughput and Latency Optimization

    Optimising TTFT, ITL and total throughput in LLM services: compute-bound vs memory-bandwidth-bound bottleneck analysis, FlashAttention/FlashInfer, CUDA Graphs, speculative decoding, parallelism tuning and SLO-driven load testing.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • Running AI and LLM Workloads on Kubernetes

    Operating GPU and LLM workloads on Kubernetes: NVIDIA GPU Operator, GPU sharing with MIG and time-slicing, queue management with Kueue/Volcano, fast model weight caching, GPU autoscaling and DCGM observability.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • Deploying Small Language Models on Edge and On-Device Hardware

    Running Small Language Models (SLMs) on edge devices, field hardware and NPUs/embedded GPUs: Llama 3.2, Qwen, Gemma and Phi (1B–4B), ONNX Runtime GenAI, ExecuTorch, TensorRT, Jetson and offline field scenarios.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • LLM Unit Economics: Cost per Request and Token Optimization

    Unit cost analysis and optimisation in LLM systems: self-hosted GPU vs commercial API break-even math, prompt and output token reduction, batch APIs, model cascading and AI FinOps metrics.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • LLM Model Serving and Deployment Fundamentals

    Shows how an open-weight LLM is served as an OpenAI-compatible API on a GPU server: inference engines, VRAM math, core serving settings and a first benchmark. An entry-level programme for developer and platform teams.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • LLM Serving with vLLM: Production Inference Deployment

    High-throughput LLM serving with vLLM: PagedAttention, continuous batching, tensor parallelism, prefix caching, multi-LoRA serving and Prometheus metrics. Teams leave with a production-ready vLLM deployment and benchmark report.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View

Get in touch