Model Online / On-Site
Duration
Modules 5 modules

Course Objective & Scope

Without a solid baseline, teams can spend weeks fine-tuning models only to discover that a well-structured few-shot prompt performs just as well. This course teaches how to build a rigorous measurement harness and acceptance framework before running a single training job.

Hands-on labs cover curating leakage-free golden evaluation sets, combining deterministic checks (exact match, F1, JSON schema validation, execution accuracy) with calibrated LLM-as-a-judge rubrics, measuring human-judge agreement and setting latency/cost guardrails. Participants leave with a working baseline benchmark pipeline and acceptance spec for their own task.

Who Should Attend?

  • Data scientists and AI engineers running model adaptation or prompt/RAG optimisation
  • Evaluation and QA engineers validating AI output quality
  • Technical product managers and leads defining project acceptance criteria
  • Prerequisite: Python and basic statistics/model evaluation familiarity

Key Takeaways

  • Construct a leakage-free golden test set covering realistic distributions and adversarial edge cases
  • Establish zero-shot and few-shot baselines across frontier reference models and target small models
  • Translate business goals into deterministic code graders and rubric-based LLM judges
  • Measure and calibrate judge alignment against human domain experts (Cohen's kappa)
  • Define concrete quality, latency and unit-cost acceptance thresholds for shipping a custom model

Curriculum

Why Baseline and Evals Come First
  • Evaluation-driven development: measure before you adapt
  • Setting the ceiling (frontier model + few-shot) and the floor (small zero-shot model)
Golden Test Set Design and Stratification
  • Stratifying typical queries, long-tail edge cases and out-of-scope/refusal prompts
  • Enforcing strict train-test splits and versioning to prevent contamination
Task-Specific Metric Engineering
  • Deterministic graders: classification F1, JSON schema adherence and functional execution
  • Binary (pass/fail) and analytic rubrics instead of vague 1-to-5 Likert scales
  • Non-negotiable operational metrics: TTFT, p95 latency and cost per request
LLM-as-a-Judge Calibration and Statistical Confidence
  • Mitigating position bias, verbosity bias and self-preference in LLM judges
  • Measuring human-judge agreement and computing bootstrap confidence intervals
Workshop: Building the Baseline Benchmark Pipeline
  • Coding an automated baseline runner and comparison leaderboard for an internal task
  • Defining the formal go-live acceptance gate specification

Other Courses in This Category

  • Fine-Tuning Fundamentals: Introduction to Customising LLMs

    Foundational hands-on course on how LLM fine-tuning works: dataset formatting, chat templates, running a first adapter training job with Hugging Face TRL and Unsloth, and avoiding overfitting.

    LLM Customisation & Model Adaptation Corporate
    Live Course
    5 modules
    View
  • Advanced Fine-Tuning: Continual Pre-Training, Long Context and Model Merging

    Advanced course covering continual pre-training (CPT), RoPE long-context extension, multi-turn tool-calling fine-tuning, catastrophic forgetting mitigation and model merging (SLERP, TIES, DARE) with MergeKit.

    LLM Customisation & Model Adaptation Corporate
    Live Course
    5 modules
    View
  • Supervised Fine-Tuning (SFT): Engineering and Optimisation

    Deep-dive engineering course on Supervised Fine-Tuning (SFT): completion-only loss masking, sequence packing, FlashAttention-2, NEFTune, learning rate schedules and production workflows with TRL SFTTrainer and Axolotl.

    LLM Customisation & Model Adaptation Corporate
    Live Course
    5 modules
    View
  • Parameter-Efficient Fine-Tuning (PEFT): Architectures and Multi-Adapter Serving

    Course on PEFT architectures—LoRA, DoRA, rsLoRA, VeRA, LoftQ, IA3 and Prefix Tuning—VRAM memory budgeting and serving dozens of dynamic adapters on a single base model with vLLM and SGLang.

    LLM Customisation & Model Adaptation Corporate
    Live Course
    5 modules
    View
  • Practical LLM Fine-Tuning with LoRA and QLoRA

    Hands-on training on Low-Rank Adaptation (LoRA) and 4-bit NormalFloat (NF4) QLoRA: rank/alpha tuning, target modules, double quantisation, paged optimizers and safe adapter merging.

    LLM Customisation & Model Adaptation Corporate
    Live Course
    5 modules
    View
  • Instruction Tuning: Following Complex Instructions, Schemas and Domain Constraints

    Course on designing and running instruction tuning so models reliably follow complex system prompts, strict JSON/schema constraints, corporate tone and refusal boundaries.

    LLM Customisation & Model Adaptation Corporate
    Live Course
    5 modules
    View

Get in touch