Model Online / On-Site
Duration
Modules 5 modules

Course Objective & Scope

The biggest leap in coding, SQL generation, quantitative analysis and rule-heavy domain reasoning comes from Reinforcement Learning with Verifiable Rewards (RLVR), as pioneered by DeepSeek-R1. Group Relative Policy Optimization (GRPO) removes PPO's memory-heavy critic model and computes advantages across sampled completion groups, making reasoning RL practical on enterprise hardware.

This advanced course covers GRPO mathematics, designing deterministic rule and execution verifiers, preventing reward hacking, cold-start SFT warmups, and high-throughput rollout training with TRL GRPOTrainer, Unsloth and vLLM. Participants train a domain reasoning model with GRPO on a verifiable enterprise task.

Who Should Attend?

  • Senior AI engineers building models for complex reasoning, code/SQL generation and policy compliance
  • Machine learning researchers and data scientists working on post-training and RL pipelines
  • Teams aiming to instil strong domain reasoning into compact open-weight models
  • Prerequisite: PyTorch, prior SFT/DPO experience and familiarity with optimization fundamentals

Key Takeaways

  • Explain the mathematical and memory advantages of critic-free GRPO over standard PPO
  • Write deterministic verifiable reward functions (RLVR) using code execution, SQL engines, constraint solvers and regex/schema checks
  • Combine format rewards and correctness rewards without triggering reward hacking or length collapse
  • Run fast GRPO training loops using vLLM-accelerated rollouts with TRL GRPOTrainer and Unsloth
  • Monitor completion length evolution, group reward variance and KL divergence to ensure stable convergence

Curriculum

From RLHF to RLVR and from PPO to GRPO
  • Replacing neural reward models with deterministic Verifiable Rewards (RLVR)
  • How GRPO eliminates the PPO critic model by normalising rewards across $G$ group rollouts
  • The clipped surrogate objective, group advantage computation and per-token KL penalties
Designing Verifiable Reward Functions and Environments
  • Writing deterministic verifiers for math, SQL execution, unit tests, JSON schemas and business rules
  • Combining structural format rewards (<think>...</think>) with outcome and partial-credit rewards
  • Filtering out zero-variance prompts (all-pass or all-fail groups) and difficulty curriculum design
Cold-Start SFT and Multi-Stage Reasoning Pipelines
  • Pure RL from base/instruct checkpoints versus Cold-Start SFT warmup on curated reasoning traces
  • Preventing reward hacking, language mixing and runaway chain-of-thought verbosity
High-Throughput Training with TRL GRPOTrainer, Unsloth and vLLM
  • Why autoregressive rollout generation dominates RL wall-clock time and how vLLM solves it
  • Running memory-efficient GRPO with LoRA/QLoRA, Unsloth and vLLM sleep/wake offloading
  • Modern refinements (Dr. GRPO, DAPO): removing length bias and preventing entropy collapse
Workshop: Training a Domain Reasoning Model with GRPO
  • Implementing a multi-component verifier and running GRPO on an enterprise SQL or rule-reasoning task
  • Evaluating pass@1 and majority-vote accuracy gains over the SFT baseline

Other Courses in This Category

  • Dataset Quality and Decontamination for LLM Training

    Course on auditing and cleaning LLM training datasets: exact and MinHash/semantic deduplication, train-test and benchmark decontamination, perplexity/judge quality filtering and PII scrubbing.

    LLM Customisation & Model Adaptation Corporate
    Live Course
    5 modules
    View
  • Prompting, RAG or Fine-Tuning: LLM Adaptation Decision Framework

    Hands-on course on choosing and combining context engineering, RAG and fine-tuning based on knowledge freshness, style/behaviour alignment, latency and TCO, producing an adaptation decision matrix.

    LLM Customisation & Model Adaptation Corporate
    Live Course
    5 modules
    View
  • Model and Data Readiness: Base Model Selection and Data Suitability

    Course on selecting open-weight or commercial base models by size, licence, tokenizer and hardware footprint, and auditing enterprise data volume, quality and privacy readiness before adaptation.

    LLM Customisation & Model Adaptation Corporate
    Live Course
    5 modules
    View
  • Quality, Cost and Risk Analysis in LLM Adaptation

    Course on modelling total cost of ownership (data curation, training and 24/7 serving), quality-versus-regression trade-offs and operational/regulatory risks for LLM customisation projects.

    LLM Customisation & Model Adaptation Corporate
    Live Course
    5 modules
    View
  • Baselines and Success Metrics: Pre-Adaptation Benchmarking for LLMs

    Course on building golden test sets, establishing zero/few-shot baselines, defining deterministic and LLM-as-a-judge metrics and setting acceptance thresholds before starting model adaptation.

    LLM Customisation & Model Adaptation Corporate
    Live Course
    5 modules
    View
  • Fine-Tuning Fundamentals: Introduction to Customising LLMs

    Foundational hands-on course on how LLM fine-tuning works: dataset formatting, chat templates, running a first adapter training job with Hugging Face TRL and Unsloth, and avoiding overfitting.

    LLM Customisation & Model Adaptation Corporate
    Live Course
    5 modules
    View

Get in touch