Model Online / On-Site
Duration
Modules 5 modules

Course Objective & Scope

In dynamic quantization weights are converted to lower precision ahead of time while activation scales are computed at inference time, delivering quick gains without preparing a calibration dataset. The programme examines the mathematical difference between dynamic and static quantization, per-tensor and per-token scaling strategies, and which layers (Linear, LSTM, attention projections) benefit most.

In the exercises BERT-style encoder models are accelerated on CPU with PyTorch torch.ao.quantization and ONNX Runtime, and an LLM is served in vLLM with FP8 dynamic quantization. Results are compared with static methods in terms of latency, accuracy and memory.

Who Should Attend?

  • ML and deep learning engineers doing model optimisation
  • Platform teams running inference on CPUs or limited GPUs
  • MLOps engineers deploying embedding, classification and LLM models
  • Prerequisite: PyTorch experience and basic quantization concepts

Key Takeaways

  • Explain the differences between dynamic, static and weight-only quantization
  • Apply dynamic quantization with PyTorch and ONNX Runtime
  • Configure FP8 dynamic quantization for LLMs
  • Run layer-wise sensitivity analysis and keep critical layers in higher precision
  • Select a method using latency, accuracy and memory measurements

Curriculum

Dynamic versus Static Quantization
  • Computing scale and zero-point, runtime scaling
  • Per-tensor, per-channel and per-token approaches
Dynamic Quantization with PyTorch
  • torch.ao.quantization and supported layers
  • CPU acceleration of encoder models
ONNX Runtime and Portable Models
  • ONNX export and dynamic INT8 quantization
  • Latency measurement on different hardware
FP8 Dynamic Quantization for LLMs
  • FP8 dynamic scaling with vLLM and LLM Compressor
  • Combining with KV cache quantization
Sensitivity Analysis and Decision
  • Layer-wise error analysis and mixed precision
  • Comparative report against static methods

Other Courses in This Category

  • Quantization-Aware Training, QLoRA Fine-Tuning and Quantized Serving

    Covers quantization end to end from training to serving: 4-bit fine-tuning with QLoRA, quantization-aware training (QAT), merging adapters, re-quantizing and serving in vLLM. For teams adapting models to company data on a single GPU.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • GPU Capacity Planning and Hardware Sizing for LLM Inference

    GPU capacity planning for LLM workloads: deriving VRAM, GPU count and server needs from model size, context length, concurrent users and SLO targets, plus a cost comparison of H100/H200/B200, L40S and cloud options.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • In-House LLM Platform Architecture: From Hardware Sizing to Air-Gapped Production

    End-to-end in-house LLM platform design for data residency and regulated environments: GPU sizing, Kubernetes cluster, vLLM/Triton serving, model gateway, security and air-gapped production architecture. Built for platform and infrastructure teams.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • Air-Gapped LLM Deployment, Offline Registry Mirroring and Update Operations

    Deploying and updating LLM services in air-gapped networks: offline container, PyPI and Hugging Face mirroring, safetensors and SBOM verification, GPU driver bundling and zero-downtime patching. Built for security-focused platform teams.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • Running Local LLMs on Desktop and Workstations: llama.cpp, GGUF and LM Studio

    Running local LLMs on desktops and workstations with llama.cpp, GGUF and LM Studio: k-quant selection, CPU+GPU layer offloading, Metal/CUDA/Vulkan acceleration and local OpenAI-compatible APIs. For developers and analysts.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • Self-Hosted AI Coding Assistants with Local LLMs

    Building self-hosted AI coding assistants without sending source code outside the company: Qwen2.5-Coder and DeepSeek-Coder models, FIM autocomplete, Continue.dev/Tabby IDE plugins, vLLM serving and codebase RAG integration.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View

Get in touch