Model Online / On-Site
Duration
Modules 5 modules

Course Objective & Scope

A lower validation loss during fine-tuning does not guarantee that the model performs better on real user tasks—or that its general reasoning and safety guardrails survived training intact. This course teaches how to evaluate custom checkpoints across task accuracy, capability retention and safety before promotion to production.

Participants use lm-evaluation-harness and LightEval for capability regression testing, build domain-specific deterministic and LLM-as-a-judge evaluators, run blinded pairwise win-rate comparisons and test for safety erosion. The deliverable is an automated Fine-Tuning Eval pipeline and Model Comparison Report generator.

Who Should Attend?

  • Data scientists and AI engineers evaluating fine-tuning runs and selecting production checkpoints
  • ML and QA engineers responsible for model governance and quality assurance
  • MLOps teams automating post-training evaluation gates
  • Prerequisite: Python and foundational LLM / fine-tuning knowledge

Key Takeaways

  • Bridge the gap between token perplexity and downstream task quality with generative evaluations
  • Author and run custom domain tasks inside lm-evaluation-harness
  • Quantify catastrophic forgetting across general reasoning, coding and language benchmarks after adaptation
  • Compare base, SFT and DPO checkpoints via blinded pairwise win rates and Elo rankings
  • Generate automated Model Cards and checkpoint comparison reports for release sign-off

Curriculum

Anatomy of Fine-Tuning Evaluation
  • Why validation loss misleads: teacher-forced perplexity versus free autoregressive generation
  • The three evaluation pillars: target task lift, general capability retention and safety alignment
Standard Benchmarks and Regression Testing (lm-eval-harness)
  • Log-likelihood multiple-choice versus generative evaluation mechanics
  • Adding custom enterprise YAML tasks to lm-evaluation-harness
  • High-throughput checkpoint evaluation backed by vLLM
Pairwise Comparison and Checkpoint Selection
  • Selecting the optimal checkpoint across epochs beyond validation loss minima
  • Position-swapped pairwise LLM-as-a-judge and Bradley-Terry / Elo scoring
  • Flip analysis: inspecting examples improved versus regressed by fine-tuning
Faithfulness, Memorisation and Safety Audits
  • Testing for verbatim training-data memorisation and canary extraction
  • Checking for post-SFT safety guardrail erosion and over-refusal drift
Workshop: Model Comparison Report and Release Gate
  • Running a multi-dimensional evaluation across a base model and two fine-tuned checkpoints
  • Producing a standardised Model Comparison Report for production sign-off

Other Courses in This Category

Get in touch