Model Online / On-Site
Duration
Modules 5 modules

Course Objective & Scope

In LLM fine-tuning, 1,000 carefully structured, consistent and diverse examples regularly outperform 100,000 noisy, unbalanced production logs. This course teaches how to architect fine-tuning datasets and run human-in-the-loop curation workflows that directly shape model behaviour.

Labs cover single- and multi-turn conversation schema design, task and difficulty mixture balancing, writing unambiguous SME annotation playbooks, measuring inter-annotator agreement, working with Argilla/Label Studio and closing the production data flywheel. Participants leave with a concrete fine-tuning dataset draft and annotation specification for their organisation.

Who Should Attend?

  • Data scientists and AI engineers curating datasets for model adaptation
  • AI product and data leads coordinating domain-expert (SME) annotation programmes
  • Data engineers building enterprise LLM data pipelines
  • Prerequisite: Python, JSON/JSONL data structures and basic LLM familiarity

Key Takeaways

  • Define clean training schemas structuring system instructions, retrieved context, reasoning steps and target outputs
  • Balance dataset mixtures across sub-tasks, input lengths, languages and refusal edge cases
  • Write unambiguous annotation guidelines and gold reference examples for domain experts
  • Measure inter-annotator agreement (Cohen's/Fleiss' kappa) to catch and fix ambiguous labelling rules
  • Set up a production data flywheel that converts corrected model failures into versioned dataset releases

Curriculum

Data-Centric Fine-Tuning: The LIMA Principle and Schema Design
  • Why stylistic consistency and label quality outweigh raw row count
  • Standardising system, user, context, tool-call and assistant fields
  • Deciding when to include explicit scratchpad or reasoning fields in targets
Data Mixing Ratios and Distribution Balancing
  • Choosing mixture ratios across primary tasks, auxiliary tasks and general replay data
  • Balancing response length, tone, complexity and abstention/out-of-scope examples
  • Temperature-based sampling and upsampling rare enterprise edge cases
SME Annotation Workflows and Quality Control
  • Writing an annotation playbook with clear rubrics and boundary examples for domain experts
  • Accelerating curation with LLM pre-labelling plus human editing in Argilla or Label Studio
  • Measuring inter-annotator agreement (IAA) and adjudicating disputed examples
Dataset Versioning, Lineage and the Data Flywheel
  • Linking every model checkpoint to an immutable dataset version and provenance manifest
  • Active-learning loops that harvest hard failures and user corrections from production
Workshop: Fine-Tuning Dataset Blueprint and Playbook
  • Drafting the schema, mixture plan and seed gold examples for an organisational use case
  • Auditing a sample dataset for formatting inconsistencies and coverage gaps

Other Courses in This Category

  • Model Evaluation and Fine-Tuning Evals for Custom LLMs

    Course on evaluating fine-tuned LLMs for task accuracy, catastrophic forgetting regressions, hallucination and safety drift using lm-evaluation-harness, custom task suites and pairwise arena comparisons.

    LLM Customisation & Model Adaptation Corporate
    Live Course
    5 modules
    View
  • Open-Weight Model Adaptation: Customising Llama, Qwen, Mistral and Gemma

    Course on adapting open-weight models—Llama, Qwen, Mistral, Gemma and DeepSeek—for enterprise use: architectural differences, custom tokens, post-training quantisation (AWQ/GGUF/FP8) and self-hosted serving.

    LLM Customisation & Model Adaptation Corporate
    Live Course
    5 modules
    View
  • Turkish Language Models: Tokenization, Evaluation and Adaptation

    Course on adapting LLMs for Turkish: morphological challenges in BPE tokenization, vocabulary extension, continual pre-training, Turkish evaluation benchmarks and cultural/regulatory alignment.

    LLM Customisation & Model Adaptation Corporate
    Live Course
    5 modules
    View
  • Distributed LLM Training with PyTorch FSDP and DeepSpeed

    Advanced course on scaling LLM training across multi-GPU and multi-node clusters using Data, Tensor, Pipeline and Context Parallelism, DeepSpeed ZeRO-1/2/3, PyTorch FSDP2 and communication profiling.

    LLM Customisation & Model Adaptation Corporate
    Live Course
    5 modules
    View
  • Reinforcement Learning for Reasoning Models (GRPO and RLVR)

    Advanced hands-on course on training DeepSeek-R1-style reasoning models using Reinforcement Learning with Verifiable Rewards (RLVR) and critic-free Group Relative Policy Optimization (GRPO).

    LLM Customisation & Model Adaptation Corporate
    Live Course
    5 modules
    View
  • Dataset Quality and Decontamination for LLM Training

    Course on auditing and cleaning LLM training datasets: exact and MinHash/semantic deduplication, train-test and benchmark decontamination, perplexity/judge quality filtering and PII scrubbing.

    LLM Customisation & Model Adaptation Corporate
    Live Course
    5 modules
    View

Get in touch