Course Objective & Scope
Testing a new prompt, RAG strategy or foundation model in live traffic is very different from A/B testing a checkout button: AI outputs are stochastic, users adapt their prompting over time, first-week novelty inflates engagement, and quality gains often trade off against latency or cost. This course teaches rigorous, practical experimentation for AI products.
Participants design an experimentation funnel spanning offline eval gates, shadow deployments, side-by-side/interleaved comparisons and online A/B tests, producing a complete experiment and ship-decision plan for their product.
Who Should Attend?
- Product managers (PMs), growth teams and experimentation platform specialists
- Data scientists, product analysts and AI engineers
- Product and engineering leaders governing release decisions
- Prerequisite: basic familiarity with A/B testing or product analytics
Key Takeaways
- Filter candidate prompts and models rapidly with offline evals and historical trace replay before running live tests
- Run risk-free shadow deployments on live production traffic
- Select the right randomisation unit (user, session or request) and avoid carryover bias
- Control for novelty effects, power-user skew and latency confounders
- Define multi-objective ship/rollback criteria across quality, conversion, latency and unit cost
Curriculum
The AI Experimentation Funnel: From Offline Evals to Live Traffic
- Why you cannot wait weeks of A/B testing for every prompt or model tweak
- Staged filtering: golden-set evals → trace replay → shadow mode → online A/B test
Shadow Deployments and Pairwise / Interleaved Testing
- Routing live traffic asynchronously to candidate models and grading with LLM judges
- Using interleaving and side-by-side preference prompts for high-sensitivity comparisons
Online A/B Test Design and Statistical Pitfalls
- Choosing randomisation units (user versus thread), power analysis and variance reduction (CUPED)
- Isolating novelty spikes and user learning effects across cohorts
Multi-Objective Trade-Offs: Quality, Latency and Cost
- Combining primary success metrics with latency, safety and cost guardrails
- Slice-based analysis across query complexity, language and user tenure
Workshop: AI Experiment and Ship-Decision Plan
- Designing an end-to-end experiment for an upcoming model or feature change
- Delivering the hypothesis, sample plan, guardrail thresholds and rollout matrix
Other Courses in This Category
-
Model behaviour and quality management for product teams: behaviour specifications (tone, boundaries, refusal style), systematic error analysis, domain-expert rubrics and continuous quality loops.
- Live Course
- 5 modules
-
Managing the quality, latency and cost triangle in AI products: model routing and cascading, prompt/semantic caching, token budgeting, perceived-speed UX and gross margin protection.
- Live Course
- 5 modules
-
Managing AI-native products where AI is the core engine rather than a bolted-on feature: compound AI systems, eval-driven development lifecycles, data flywheels and modern PM–AI engineer collaboration.
- Live Course
- 5 modules
-
Agent Experience (AX) and delegation UX for products where users hand off multi-step jobs to autonomous agents: task briefing, plan sign-off, async progress tracking, blocker escalation and deliverable review.
- Live Course
- 5 modules
-
Governing foundation model migrations, prompt edits and tool schema updates in live AI products without regressions: versioning, automated eval gates, canary rollouts and rollback protocols.
- Live Course
- 5 modules
-
AI product strategy for product leaders: value proposition and defensibility (moats), build-versus-buy-versus-partner decisions, token and inference unit economics, pricing models and an actionable AI strategy brief.
- Live Course
- 5 modules
Get in touch
