Course Objective & Scope
Quantizing a model after fine-tuning it on company data can erase part of the accuracy gained. This programme treats quantization as part of training: LoRA training on a 4-bit NF4 base model with QLoRA, quantization-aware training with PyTorch torchao, and merging adapters after training before converting to AWQ, GPTQ or FP8.
Participants adapt a model to a sample business task using Hugging Face PEFT, TRL and Unsloth, then quantize it and serve it in vLLM as multi-LoRA or a merged model. Quality loss is tracked with evals at every step and the training-to-serving pipeline is documented.
Who Should Attend?
- ML engineers and data scientists doing fine-tuning
- AI teams adapting models on a limited GPU budget
- MLOps engineers moving trained models into production
- Prerequisite: PyTorch, Transformers and basic fine-tuning experience
Key Takeaways
- Fine-tune large models on a single GPU with QLoRA
- Determine when quantization-aware training is needed
- Merge LoRA adapters and re-quantize the model
- Serve the adapted model efficiently in vLLM
- Track quality loss from training to serving with evals
Curriculum
Low Precision in Training
- Mixed-precision (BF16/FP8) training and memory budget
- NF4, double quantization and paged optimizers
Fine-Tuning with QLoRA
- QLoRA training with PEFT, TRL and Unsloth
- LoRA rank, target modules and hyperparameters
Quantization-Aware Training
- Fake quantization and QAT with torchao
- Accuracy comparison of PTQ versus QAT
Merging and Re-Quantization
- Adapter merge and AWQ/GPTQ/FP8 conversion
- Options for exporting to GGUF
Serving and Quality Tracking
- Serving multi-LoRA and merged models in vLLM
- Eval comparison across the training-to-serving pipeline
Other Courses in This Category
-
GPU capacity planning for LLM workloads: deriving VRAM, GPU count and server needs from model size, context length, concurrent users and SLO targets, plus a cost comparison of H100/H200/B200, L40S and cloud options.
- Live Course
- 5 modules
-
End-to-end in-house LLM platform design for data residency and regulated environments: GPU sizing, Kubernetes cluster, vLLM/Triton serving, model gateway, security and air-gapped production architecture. Built for platform and infrastructure teams.
- Live Course
- 5 modules
-
Deploying and updating LLM services in air-gapped networks: offline container, PyPI and Hugging Face mirroring, safetensors and SBOM verification, GPU driver bundling and zero-downtime patching. Built for security-focused platform teams.
- Live Course
- 5 modules
-
Running local LLMs on desktops and workstations with llama.cpp, GGUF and LM Studio: k-quant selection, CPU+GPU layer offloading, Metal/CUDA/Vulkan acceleration and local OpenAI-compatible APIs. For developers and analysts.
- Live Course
- 5 modules
-
Building self-hosted AI coding assistants without sending source code outside the company: Qwen2.5-Coder and DeepSeek-Coder models, FIM autocomplete, Continue.dev/Tabby IDE plugins, vLLM serving and codebase RAG integration.
- Live Course
- 5 modules
-
Selecting open-weight LLMs for enterprise use: Llama, Qwen, Mistral, Gemma and DeepSeek families; commercial licence analysis, dense vs MoE trade-offs, Turkish and tool-calling capability, and building a benchmark matrix on company data.
- Live Course
- 5 modules
Get in touch
