Model Online / On-Site
Duration
Modules 5 modules

Course Objective & Scope

Near-duplicate records in training data cause models to memorise and regurgitate boilerplate, while accidental leakage of test or benchmark prompts into training splits creates illusory evaluation scores that collapse in production. This course teaches how to build automated dataset quality, deduplication and decontamination pipelines before training.

Hands-on labs cover exact and fuzzy MinHashLSH deduplication, embedding-based semantic deduplication (SemDeDup), n-gram and semantic train-test decontamination, Instruction-Following Difficulty (IFD) and judge-based quality filtering, and Turkish/multilingual PII scrubbing. Participants build an automated dataset audit pipeline on a noisy corpus.

Who Should Attend?

  • Data scientists and data engineers preparing corpora for LLM fine-tuning and pre-training
  • AI engineers responsible for benchmark integrity and evaluation reliability
  • Platform teams enforcing privacy and PII compliance on training datasets
  • Prerequisite: Python, data-processing libraries and basic familiarity with embeddings

Key Takeaways

  • Remove exact, fuzzy (MinHashLSH) and semantic (embedding cosine) duplicates from instruction and document corpora
  • Decontaminate training splits against internal golden test sets and public benchmarks using n-gram and vector overlap
  • Filter out truncated completions, encoding corruption, language drift and noisy labels automatically
  • Select high-value training subsets using Instruction-Following Difficulty (IFD) and calibrated judge scores
  • Detect and redact PII (national IDs, IBANs, emails, phone numbers) and secrets prior to model training

Curriculum

Data Noise Taxonomy and Multi-Stage Deduplication
  • Exact hashing (SHA-256), fuzzy n-gram MinHashLSH and embedding-based semantic deduplication (SemDeDup)
  • How duplicate clusters inflate memorisation and degrade generalisation
Train–Test and Benchmark Decontamination
  • Combining n-gram overlap (8–13 grams) with dense retrieval to catch paraphrased test leakage
  • Preventing teacher models from leaking public benchmark items during synthetic generation
  • Entity- and time-grouped splits to prevent indirect leakage across related records
Heuristic, Statistical and Model-Based Quality Filtering
  • Catching encoding corruption, degenerate repetition loops, truncated outputs and language drift
  • Pruning trivial or noisy samples using perplexity and Instruction-Following Difficulty (IFD) scores
  • Surfacing mislabelled training examples via high-loss inspection and judge rubrics
PII Scrubbing, Privacy and Backdoor Scanning
  • Redacting personal identifiers (IDs, IBANs, phone numbers, names) with regex, NER and Microsoft Presidio
  • Scanning training corpora for leaked credentials, prompt-injection payloads and poison triggers
Workshop: Automated Dataset Hygiene and Decontamination Pipeline
  • Cleaning a seeded dataset containing duplicates, leaked test items, PII and corrupted labels
  • Comparing model accuracy when trained on the raw corpus versus the cleaned, decontaminated subset

Other Courses in This Category

  • Prompting, RAG or Fine-Tuning: LLM Adaptation Decision Framework

    Hands-on course on choosing and combining context engineering, RAG and fine-tuning based on knowledge freshness, style/behaviour alignment, latency and TCO, producing an adaptation decision matrix.

    LLM Customisation & Model Adaptation Corporate
    Live Course
    5 modules
    View
  • Model and Data Readiness: Base Model Selection and Data Suitability

    Course on selecting open-weight or commercial base models by size, licence, tokenizer and hardware footprint, and auditing enterprise data volume, quality and privacy readiness before adaptation.

    LLM Customisation & Model Adaptation Corporate
    Live Course
    5 modules
    View
  • Quality, Cost and Risk Analysis in LLM Adaptation

    Course on modelling total cost of ownership (data curation, training and 24/7 serving), quality-versus-regression trade-offs and operational/regulatory risks for LLM customisation projects.

    LLM Customisation & Model Adaptation Corporate
    Live Course
    5 modules
    View
  • Baselines and Success Metrics: Pre-Adaptation Benchmarking for LLMs

    Course on building golden test sets, establishing zero/few-shot baselines, defining deterministic and LLM-as-a-judge metrics and setting acceptance thresholds before starting model adaptation.

    LLM Customisation & Model Adaptation Corporate
    Live Course
    5 modules
    View
  • Fine-Tuning Fundamentals: Introduction to Customising LLMs

    Foundational hands-on course on how LLM fine-tuning works: dataset formatting, chat templates, running a first adapter training job with Hugging Face TRL and Unsloth, and avoiding overfitting.

    LLM Customisation & Model Adaptation Corporate
    Live Course
    5 modules
    View
  • Advanced Fine-Tuning: Continual Pre-Training, Long Context and Model Merging

    Advanced course covering continual pre-training (CPT), RoPE long-context extension, multi-turn tool-calling fine-tuning, catastrophic forgetting mitigation and model merging (SLERP, TIES, DARE) with MergeKit.

    LLM Customisation & Model Adaptation Corporate
    Live Course
    5 modules
    View

Get in touch