Model Online / On-Site
Duration
Modules 6 modules

Course Objective & Scope

Most frontier open-weight models—including DeepSeek, Qwen, Mixtral and Llama 4—use sparse Mixture of Experts (MoE) architectures to combine high model capacity with lower compute per token. Yet MoE models create a distinct hardware trade-off: high VRAM requirements for total parameters paired with fast inference from a small subset of active parameters.

This course examines MoE layers, top-k gating and shared-expert designs, memory bandwidth bottlenecks, tensor and expert parallelism in vLLM and SGLang, and QLoRA fine-tuning on MoE models through hands-on labs.

Who Should Attend?

  • AI and MLOps engineers deploying open-weight models on enterprise infrastructure
  • Platform architects handling model selection, quantization and GPU capacity planning
  • Data scientists and researchers fine-tuning MoE models
  • Prerequisite: understanding of the Transformer architecture and basic GPU/PyTorch experience

Key Takeaways

  • Calculate total parameters, active parameters, FLOPs and VRAM needs for dense vs sparse MoE models
  • Evaluate fine-grained and shared-expert MoE designs for quality and serving implications
  • Select quantization and parallelism strategies for multi-GPU MoE deployment
  • Benchmark MoE throughput and latency across different batch sizes and traffic patterns
  • Configure LoRA/QLoRA fine-tuning targeting the right modules in an MoE model

Curriculum

From Dense Transformers to Sparse MoE
  • Replacing dense FFN layers with routed experts and top-k gating
  • Total parameters, active parameters and compute cost per token
Modern MoE Architectures and Load Balancing
  • Architectural comparison of Mixtral, Qwen-MoE, DeepSeek-V3 and Llama 4
  • Fine-grained experts, shared experts and auxiliary load-balancing losses
Hardware, VRAM and Memory Bandwidth Analysis
  • Why MoE models are fast at low concurrency yet VRAM-heavy
  • How active expert ratios and GPU utilisation shift as batch size grows
MoE Model Serving and Parallelism
  • Tensor Parallelism and Expert Parallelism in vLLM and SGLang
  • Impact of FP8, AWQ and GPTQ quantization on router stability
Fine-Tuning MoE Models with LoRA and QLoRA
  • Targeting attention projections versus expert FFN layers
  • Router freezing and preventing expert collapse during adaptation
Lab: Benchmarking Dense vs MoE Models
  • Speed and quality comparison at matching active parameter budgets
  • GPU and model sizing report for an internal workload profile

Other Courses in This Category

  • Streaming and Real-Time AI Application Engineering

    Building low-latency streaming text, partial structured output and bidirectional voice AI applications using SSE, WebSockets and WebRTC, with TTFT optimisation, barge-in handling and concurrency control.

    LLM Application Engineering Corporate
    Live Course
    6 modules
    View
  • Small Language Models (SLMs) for Enterprise Applications

    Building fast, low-cost enterprise solutions with 1B–14B Small Language Models (Llama, Qwen, Gemma, Phi, Ministral) for classification, extraction and routing, including distillation and task adaptation.

    LLM Application Engineering Corporate
    Live Course
    6 modules
    View
  • Open-Weight LLM Integration and Self-Hosted Serving

    Deploying and integrating open-weight LLMs (Llama, Qwen, Mistral, DeepSeek) on-premises or in private cloud as OpenAI-compatible APIs using vLLM, SGLang and Ollama, including GPU sizing and quantization.

    LLM Application Engineering Corporate
    Live Course
    6 modules
    View
  • LLM Cost and Latency Optimisation: Prompt Caching, Batching and FinOps

    Engineering techniques to cut token spend and p95 latency in production LLM applications: prompt and semantic caching, Batch APIs, output length control, tiered models and unit economics.

    LLM Application Engineering Corporate
    Live Course
    6 modules
    View
  • LangChain Fundamentals for LLM Application Development

    Hands-on programme on building modular LLM applications with modern LangChain (LCEL, ChatModel abstractions, prompt templates, structured outputs, tool calling, retrievers and LangSmith tracing).

    LLM Application Engineering Corporate
    Live Course
    6 modules
    View
  • LLM Engineering for Java & .NET (Spring AI, LangChain4j, Semantic Kernel)

    Building type-safe LLM, RAG, tool-calling and MCP integrations directly in enterprise Java (Spring AI, LangChain4j) and .NET (Microsoft.Extensions.AI, Semantic Kernel) stacks without switching to Python.

    LLM Application Engineering Corporate
    Live Course
    6 modules
    View

Get in touch