Model Online / On-Site
Duration
Modules 5 modules

Course Objective & Scope

In long-context, multi-user LLM applications, the primary consumer of GPU memory is often not the model weights but the KV cache that grows with every active request. This advanced programme covers KV cache math across MHA, GQA and MLA attention designs, PagedAttention and RadixAttention algorithms, and the progression from static batching to continuous batching and chunked prefill.

Participants test RAG and multi-turn agent workloads with long system prompts on vLLM and SGLang, combining prefix caching, FP8 KV cache quantization and scheduler tuning to multiply concurrent request capacity on the same GPU.

Who Should Attend?

  • AI and platform engineers optimising LLM inference engines in production
  • ML engineers building long-document RAG and multi-step agent infrastructure
  • MLOps teams looking to increase concurrent users per GPU
  • Prerequisite: Transformer architecture basics and hands-on experience with vLLM/TGI/SGLang

Key Takeaways

  • Calculate per-token KV cache memory footprint across MHA, GQA and MLA architectures
  • Eliminate memory fragmentation using PagedAttention and block-level allocation
  • Structure prompts to maximise hit rates in automatic prefix caching and RadixAttention
  • Apply FP8 KV cache quantization to expand maximum concurrency
  • Balance prefill and decode phases using continuous batching, max-num-batched-tokens and chunked prefill

Curriculum

Anatomy of the KV Cache and Attention Designs
  • Role of K and V matrices during prefill and decode phases
  • Memory comparison of MHA, MQA, GQA, sliding window and DeepSeek MLA
PagedAttention and Block Memory Management
  • Internal and external memory fragmentation and virtual block tables
  • Preemption (recompute versus swap) and KV cache occupancy metrics
Prefix Caching and RadixAttention
  • How vLLM Automatic Prefix Caching (APC) and SGLang RadixAttention work
  • Designing system prompts, few-shot examples and multi-turn history for high cache hit rates
KV Cache Compression and Offloading
  • FP8 KV cache quantization and its accuracy/capacity impact
  • CPU RAM/NVMe KV cache offloading and LMCache approaches
Continuous Batching and Chunked Prefill
  • Iteration-level scheduling, max-num-seqs and max-num-batched-tokens
  • Chunked prefill and disaggregated serving to prevent long prompts from stalling decode

Other Courses in This Category

Get in touch