Model Online / On-Site
Duration
Modules 5 modules

Course Objective & Scope

Testing a new prompt, RAG strategy or foundation model in live traffic is very different from A/B testing a checkout button: AI outputs are stochastic, users adapt their prompting over time, first-week novelty inflates engagement, and quality gains often trade off against latency or cost. This course teaches rigorous, practical experimentation for AI products.

Participants design an experimentation funnel spanning offline eval gates, shadow deployments, side-by-side/interleaved comparisons and online A/B tests, producing a complete experiment and ship-decision plan for their product.

Who Should Attend?

  • Product managers (PMs), growth teams and experimentation platform specialists
  • Data scientists, product analysts and AI engineers
  • Product and engineering leaders governing release decisions
  • Prerequisite: basic familiarity with A/B testing or product analytics

Key Takeaways

  • Filter candidate prompts and models rapidly with offline evals and historical trace replay before running live tests
  • Run risk-free shadow deployments on live production traffic
  • Select the right randomisation unit (user, session or request) and avoid carryover bias
  • Control for novelty effects, power-user skew and latency confounders
  • Define multi-objective ship/rollback criteria across quality, conversion, latency and unit cost

Curriculum

The AI Experimentation Funnel: From Offline Evals to Live Traffic
  • Why you cannot wait weeks of A/B testing for every prompt or model tweak
  • Staged filtering: golden-set evals → trace replay → shadow mode → online A/B test
Shadow Deployments and Pairwise / Interleaved Testing
  • Routing live traffic asynchronously to candidate models and grading with LLM judges
  • Using interleaving and side-by-side preference prompts for high-sensitivity comparisons
Online A/B Test Design and Statistical Pitfalls
  • Choosing randomisation units (user versus thread), power analysis and variance reduction (CUPED)
  • Isolating novelty spikes and user learning effects across cohorts
Multi-Objective Trade-Offs: Quality, Latency and Cost
  • Combining primary success metrics with latency, safety and cost guardrails
  • Slice-based analysis across query complexity, language and user tenure
Workshop: AI Experiment and Ship-Decision Plan
  • Designing an end-to-end experiment for an upcoming model or feature change
  • Delivering the hypothesis, sample plan, guardrail thresholds and rollout matrix

Other Courses in This Category

Get in touch