Model Online / On-Site
Duration
Modules 5 modules

Course Objective & Scope

Quantization is the most effective way to run the same model with less GPU memory and higher throughput, yet the wrong method can cause unnoticed losses in accuracy, non-English performance or tool-calling ability. The programme compares number formats (FP32, FP16, BF16, FP8, INT8, INT4), weight-only versus weight-and-activation quantization, and methods such as GPTQ, AWQ, SmoothQuant and GGUF k-quants.

In the labs participants convert the same model into different formats, run them on vLLM and llama.cpp, measure quality with a small eval set built from organisational data and produce a cost/performance comparison table.

Who Should Attend?

  • AI and platform engineers serving LLMs on limited GPUs
  • ML engineers responsible for model selection and optimisation
  • Technical leads aiming to cut in-house LLM costs
  • Prerequisite: Python and PyTorch/Transformers basics

Key Takeaways

  • Explain the memory and accuracy impact of precision formats
  • Quantize models with GPTQ, AWQ, FP8 and GGUF
  • Serve quantized models on vLLM and llama.cpp
  • Measure quality loss with perplexity and task-based evals
  • Choose a format based on hardware and use case

Curriculum

Number Formats and Memory Math
  • Comparing FP32, FP16, BF16, FP8, INT8 and INT4
  • Weight, activation and KV cache memory
Post-Training Quantization Methods
  • GPTQ, AWQ and SmoothQuant
  • Calibration dataset selection and per-channel/group-wise quantization
Tools and Formats
  • bitsandbytes, LLM Compressor and GGUF conversion
  • Evaluating ready-made quantized checkpoints
Serving Quantized Models
  • AWQ/GPTQ/FP8 in vLLM, GGUF in llama.cpp
  • Hardware support: Hopper/Blackwell FP8, consumer GPUs
Measuring Quality and Performance
  • Perplexity, task-based evals and Turkish-language tests
  • VRAM, tokens/s and cost comparison table

Other Courses in This Category

  • Dynamic Quantization for Neural Networks and LLMs

    Accelerating models with dynamic quantization without calibration data: PyTorch and ONNX Runtime dynamic quantization, FP8 dynamic activation scaling, comparison with static methods and measurement on CPU and GPU.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • Quantization-Aware Training, QLoRA Fine-Tuning and Quantized Serving

    Covers quantization end to end from training to serving: 4-bit fine-tuning with QLoRA, quantization-aware training (QAT), merging adapters, re-quantizing and serving in vLLM. For teams adapting models to company data on a single GPU.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • GPU Capacity Planning and Hardware Sizing for LLM Inference

    GPU capacity planning for LLM workloads: deriving VRAM, GPU count and server needs from model size, context length, concurrent users and SLO targets, plus a cost comparison of H100/H200/B200, L40S and cloud options.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • In-House LLM Platform Architecture: From Hardware Sizing to Air-Gapped Production

    End-to-end in-house LLM platform design for data residency and regulated environments: GPU sizing, Kubernetes cluster, vLLM/Triton serving, model gateway, security and air-gapped production architecture. Built for platform and infrastructure teams.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • Air-Gapped LLM Deployment, Offline Registry Mirroring and Update Operations

    Deploying and updating LLM services in air-gapped networks: offline container, PyPI and Hugging Face mirroring, safetensors and SBOM verification, GPU driver bundling and zero-downtime patching. Built for security-focused platform teams.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View
  • Running Local LLMs on Desktop and Workstations: llama.cpp, GGUF and LM Studio

    Running local LLMs on desktops and workstations with llama.cpp, GGUF and LM Studio: k-quant selection, CPU+GPU layer offloading, Metal/CUDA/Vulkan acceleration and local OpenAI-compatible APIs. For developers and analysts.

    Local & On-Premise LLM Platforms Corporate
    Live Course
    5 modules
    View

Get in touch