Model Online / On-Site
Duration
Modules 5 modules

Course Objective & Scope

vLLM is one of the most widely used inference engines for serving open-weight models in-house, yet with default settings a large share of GPU capacity can go to waste. The programme covers how PagedAttention and continuous batching work, critical parameters such as gpu-memory-utilization, max-model-len and max-num-seqs, and distributing large models across GPUs with tensor and pipeline parallelism.

In the labs participants configure prefix caching, speculative decoding, multi-LoRA serving, structured output and tool calling, and measure results with vLLM's own benchmark tools. Each team produces a tuned vLLM deployment and performance report for its own model.

Who Should Attend?

  • AI and platform engineers building an in-house LLM platform
  • MLOps/LLMOps teams responsible for model serving
  • Developers moving from Ollama to production scale
  • Prerequisite: Docker, Linux and basic LLM inference concepts

Key Takeaways

  • Run vLLM on Docker or Kubernetes behind an OpenAI-compatible API
  • Tune memory and batching parameters for the workload
  • Spread large models across GPUs with tensor parallelism
  • Serve multiple LoRA adapters on a single base model
  • Monitor throughput, latency and KV cache usage with Prometheus metrics

Curriculum

vLLM Architecture
  • PagedAttention, continuous batching and the scheduler
  • V1 engine, supported models and hardware
Installation and Core Configuration
  • Docker image, vllm serve and OpenAI-compatible endpoints
  • Memory, context length and concurrency parameters
Multi-GPU and Large Models
  • Tensor and pipeline parallelism
  • MoE models and quantized checkpoints (AWQ, GPTQ, FP8)
Advanced Features
  • Prefix caching, speculative decoding and chunked prefill
  • Multi-LoRA serving, structured output and tool calling
Monitoring and Benchmarking
  • vLLM metrics with Prometheus/Grafana
  • Load testing and reporting with vLLM benchmark tools

Other Courses in This Category

  • Hugging Face Text Generation Inference (TGI) for LLM Serving

    Serving LLMs with Hugging Face Text Generation Inference (TGI): Docker setup, sharding, quantization, Messages API, structured output with guidance and Inference Endpoints. A hands-on programme for teams in the Hugging Face ecosystem.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • Running Local LLMs with Ollama

    Running open-weight models on laptops, workstations or servers with Ollama: model library, Modelfile, REST/OpenAI-compatible API, Open WebUI, embeddings and a shared team setup. For fast prototyping and confidential-data scenarios.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • LLM Serving Stack Mastery: FastAPI, vLLM, KServe and Triton

    Combines a FastAPI model API layer, vLLM for LLM inference, KServe for model management on Kubernetes and Triton for multi-framework serving in one architecture. Participants build an end-to-end, scalable model serving stack.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    6 modules
    View
  • LLM Quantization: INT8, INT4, FP8, FP16 and BF16 in Practice

    FP16/BF16, FP8, INT8 and INT4 precision options for LLMs; post-training quantization with GPTQ, AWQ, GGUF and bitsandbytes, measuring quality loss with evals and verifying VRAM and throughput gains through benchmarks.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • Dynamic Quantization for Neural Networks and LLMs

    Accelerating models with dynamic quantization without calibration data: PyTorch and ONNX Runtime dynamic quantization, FP8 dynamic activation scaling, comparison with static methods and measurement on CPU and GPU.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • Quantization-Aware Training, QLoRA Fine-Tuning and Quantized Serving

    Covers quantization end to end from training to serving: 4-bit fine-tuning with QLoRA, quantization-aware training (QAT), merging adapters, re-quantizing and serving in vLLM. For teams adapting models to company data on a single GPU.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View

Get in touch