Model Online / On-Site
Duration
Modules 5 modules

Course Objective & Scope

A lower validation loss during fine-tuning does not guarantee that the model performs better on real user tasks—or that its general reasoning and safety guardrails survived training intact. This course teaches how to evaluate custom checkpoints across task accuracy, capability retention and safety before promotion to production.

Participants use lm-evaluation-harness and LightEval for capability regression testing, build domain-specific deterministic and LLM-as-a-judge evaluators, run blinded pairwise win-rate comparisons and test for safety erosion. The deliverable is an automated Fine-Tuning Eval pipeline and Model Comparison Report generator.

Who Should Attend?

  • Data scientists and AI engineers evaluating fine-tuning runs and selecting production checkpoints
  • ML and QA engineers responsible for model governance and quality assurance
  • MLOps teams automating post-training evaluation gates
  • Prerequisite: Python and foundational LLM / fine-tuning knowledge

Key Takeaways

  • Bridge the gap between token perplexity and downstream task quality with generative evaluations
  • Author and run custom domain tasks inside lm-evaluation-harness
  • Quantify catastrophic forgetting across general reasoning, coding and language benchmarks after adaptation
  • Compare base, SFT and DPO checkpoints via blinded pairwise win rates and Elo rankings
  • Generate automated Model Cards and checkpoint comparison reports for release sign-off

Curriculum

Anatomy of Fine-Tuning Evaluation
  • Why validation loss misleads: teacher-forced perplexity versus free autoregressive generation
  • The three evaluation pillars: target task lift, general capability retention and safety alignment
Standard Benchmarks and Regression Testing (lm-eval-harness)
  • Log-likelihood multiple-choice versus generative evaluation mechanics
  • Adding custom enterprise YAML tasks to lm-evaluation-harness
  • High-throughput checkpoint evaluation backed by vLLM
Pairwise Comparison and Checkpoint Selection
  • Selecting the optimal checkpoint across epochs beyond validation loss minima
  • Position-swapped pairwise LLM-as-a-judge and Bradley-Terry / Elo scoring
  • Flip analysis: inspecting examples improved versus regressed by fine-tuning
Faithfulness, Memorisation and Safety Audits
  • Testing for verbatim training-data memorisation and canary extraction
  • Checking for post-SFT safety guardrail erosion and over-refusal drift
Workshop: Model Comparison Report and Release Gate
  • Running a multi-dimensional evaluation across a base model and two fine-tuned checkpoints
  • Producing a standardised Model Comparison Report for production sign-off

Other Courses in This Category

  • Open-Weight Model Adaptation: Customising Llama, Qwen, Mistral and Gemma

    Course on adapting open-weight models—Llama, Qwen, Mistral, Gemma and DeepSeek—for enterprise use: architectural differences, custom tokens, post-training quantisation (AWQ/GGUF/FP8) and self-hosted serving.

    LLM Customisation & Model Adaptation Corporate
    Live Course
    5 modules
    View
  • Turkish Language Models: Tokenization, Evaluation and Adaptation

    Course on adapting LLMs for Turkish: morphological challenges in BPE tokenization, vocabulary extension, continual pre-training, Turkish evaluation benchmarks and cultural/regulatory alignment.

    LLM Customisation & Model Adaptation Corporate
    Live Course
    5 modules
    View
  • Distributed LLM Training with PyTorch FSDP and DeepSpeed

    Advanced course on scaling LLM training across multi-GPU and multi-node clusters using Data, Tensor, Pipeline and Context Parallelism, DeepSpeed ZeRO-1/2/3, PyTorch FSDP2 and communication profiling.

    LLM Customisation & Model Adaptation Corporate
    Live Course
    5 modules
    View
  • Reinforcement Learning for Reasoning Models (GRPO and RLVR)

    Advanced hands-on course on training DeepSeek-R1-style reasoning models using Reinforcement Learning with Verifiable Rewards (RLVR) and critic-free Group Relative Policy Optimization (GRPO).

    LLM Customisation & Model Adaptation Corporate
    Live Course
    5 modules
    View
  • Dataset Quality and Decontamination for LLM Training

    Course on auditing and cleaning LLM training datasets: exact and MinHash/semantic deduplication, train-test and benchmark decontamination, perplexity/judge quality filtering and PII scrubbing.

    LLM Customisation & Model Adaptation Corporate
    Live Course
    5 modules
    View
  • Prompting, RAG or Fine-Tuning: LLM Adaptation Decision Framework

    Hands-on course on choosing and combining context engineering, RAG and fine-tuning based on knowledge freshness, style/behaviour alignment, latency and TCO, producing an adaptation decision matrix.

    LLM Customisation & Model Adaptation Corporate
    Live Course
    5 modules
    View

Get in touch