Model Online / On-Site
Duration
Modules 5 modules

Course Objective & Scope

Supervised Fine-Tuning teaches a model how to imitate good responses, but it cannot explicitly penalise subtle hallucinations, awkward tone or near-miss formatting failures. Preference optimisation contrasts chosen and rejected completions to push probability mass away from failure modes and align the model with human and business preferences.

This course covers the evolution from RLHF/PPO to reward-model-free alignment with DPO, IPO, unpaired KTO and reference-free ORPO and SimPO using Hugging Face TRL. Participants construct an on-policy preference dataset from their own SFT model's failure cases and run a complete alignment pipeline.

Who Should Attend?

  • AI engineers pushing model accuracy, tone and safety beyond what SFT achieves
  • Data scientists turning user feedback logs (thumbs up/down, user edits) into model improvements
  • ML engineers and researchers working on post-training and alignment pipelines
  • Prerequisite: PyTorch, Hugging Face TRL and prior Supervised Fine-Tuning (SFT) experience

Key Takeaways

  • Select the right algorithm among DPO, IPO, KTO, ORPO and SimPO for your data format and compute budget
  • Generate high-signal, on-policy chosen/rejected preference pairs from your SFT checkpoint
  • Tune beta (KL penalty) and low learning rates to prevent mode collapse and alignment tax
  • Diagnose and fix verbosity bias and likelihood displacement during preference training
  • Convert binary production feedback (thumbs up/down) directly into KTO or DPO training runs

Curriculum

Why SFT Is Not Enough: From RLHF to DPO
  • Why maximum-likelihood SFT cannot explicitly suppress plausible-sounding errors
  • The Bradley-Terry preference model and the complexity of separate reward models + PPO
  • How DPO reparameterises the reward function directly through policy-to-reference log ratios
The Alignment Family: DPO, IPO, KTO, ORPO and SimPO
  • Preventing overfitting on deterministic preferences with IPO
  • Learning from unpaired binary thumbs-up/thumbs-down signals with KTO
  • Reference-free alignment with ORPO and length-normalised SimPO
Constructing High-Signal Preference Datasets
  • On-policy versus off-policy pairs: why sampling rejections from your own SFT model matters
  • Harvesting preference pairs from user edits, deterministic verifiers and LLM judges
  • Avoiding trivial rejections that yield near-zero gradients
Training Dynamics, Hyperparameters and Pitfalls
  • Tuning beta, setting learning rates 10–100x lower than SFT and using LoRA as the reference model
  • Monitoring reward margins, accuracy and chosen/rejected log-probabilities
  • Countering verbosity bias and likelihood displacement
Workshop: Aligning an SFT Model with DPO and ORPO
  • Running preference tuning with Hugging Face TRL DPOTrainer and ORPOTrainer
  • Measuring head-to-head win rates and hallucination reduction against the SFT baseline

Other Courses in This Category

  • Synthetic Data Generation: Data Synthesis and Distillation for LLM Training

    Hands-on course on generating high-diversity SFT and preference datasets in low-resource enterprise domains using teacher LLMs, Evol-Instruct, persona matrices, Distilabel and verification filters.

    LLM Customisation & Model Adaptation Corporate
    Live Course
    5 modules
    View
  • Training Data Design: Dataset Architecture for LLM Fine-Tuning

    Hands-on course on designing high-leverage SFT and multi-turn conversation datasets: the LIMA principle, task mixture ratios, expert annotation guidelines, schema design and dataset versioning.

    LLM Customisation & Model Adaptation Corporate
    Live Course
    5 modules
    View
  • Model Evaluation and Fine-Tuning Evals for Custom LLMs

    Course on evaluating fine-tuned LLMs for task accuracy, catastrophic forgetting regressions, hallucination and safety drift using lm-evaluation-harness, custom task suites and pairwise arena comparisons.

    LLM Customisation & Model Adaptation Corporate
    Live Course
    5 modules
    View
  • Open-Weight Model Adaptation: Customising Llama, Qwen, Mistral and Gemma

    Course on adapting open-weight models—Llama, Qwen, Mistral, Gemma and DeepSeek—for enterprise use: architectural differences, custom tokens, post-training quantisation (AWQ/GGUF/FP8) and self-hosted serving.

    LLM Customisation & Model Adaptation Corporate
    Live Course
    5 modules
    View
  • Turkish Language Models: Tokenization, Evaluation and Adaptation

    Course on adapting LLMs for Turkish: morphological challenges in BPE tokenization, vocabulary extension, continual pre-training, Turkish evaluation benchmarks and cultural/regulatory alignment.

    LLM Customisation & Model Adaptation Corporate
    Live Course
    5 modules
    View
  • Distributed LLM Training with PyTorch FSDP and DeepSpeed

    Advanced course on scaling LLM training across multi-GPU and multi-node clusters using Data, Tensor, Pipeline and Context Parallelism, DeepSpeed ZeRO-1/2/3, PyTorch FSDP2 and communication profiling.

    LLM Customisation & Model Adaptation Corporate
    Live Course
    5 modules
    View

Get in touch