Course Objective & Scope
Online controlled experiments (A/B tests) are the most reliable way to know whether a new product feature, ranking algorithm or UI change genuinely improves outcomes. Yet miscalculated sample sizes, p-value peeking or broken randomisation routinely push teams into shipping the wrong changes.
This course covers testable hypotheses and metric hierarchies (OEC, driver and guardrail metrics), Minimum Detectable Effect (MDE) and power calculations, frequentist and Bayesian evaluation, Sample Ratio Mismatch (SRM), multiple-testing corrections, variance reduction with CUPED, and feature-flag/experimentation platform architecture. Participants create an experiment design template and launch-decision checklist for their own product.
Who Should Attend?
- Product managers, growth specialists and digital product teams
- Data analysts and data scientists designing and evaluating experiments
- Software engineers building feature-flagging, telemetry and experimentation infrastructure
- Prerequisite: familiarity with basic probability and descriptive statistics
Key Takeaways
- Translate business goals into an Overall Evaluation Criterion (OEC) and guardrail metrics
- Calculate MDE, statistical power and required experiment duration before launch
- Diagnose peeking bias, Sample Ratio Mismatch (SRM), novelty effects and network interference
- Apply sequential testing, Bayesian inference and CUPED variance reduction appropriately
- Establish a standard internal A/B test design template and ship/rollback decision protocol
Curriculum
Experimentation Culture, Hypotheses and Metric Hierarchies
- Causality vs correlation and the anatomy of an online controlled experiment
- Selecting North Star, Overall Evaluation Criterion (OEC), secondary and guardrail metrics
- Choosing the randomisation unit (user, session, device, page)
Statistical Design: Power Analysis and Sample Size
- Type I (alpha) and Type II (beta) errors, statistical power and MDE
- Sample size and runtime calculation for conversion and continuous (revenue, latency) metrics
- A/B/n and multivariate experiments with Bonferroni and Benjamini-Hochberg corrections
Common Pitfalls and Trustworthiness Checks
- A/A tests and diagnosing Sample Ratio Mismatch (SRM)
- The peeking problem and Always-Valid / Sequential Testing
- Simpson's paradox, survivorship bias, novelty/primacy effects and carryover interference
Advanced Techniques: CUPED, Bayesian Testing and Segmentation
- Variance reduction and shorter runtimes using pre-experiment covariates (CUPED)
- Bayesian A/B testing: probability to beat baseline and expected loss
- Heterogeneous treatment effects (HTE) and disciplined subgroup analysis
Experimentation Infrastructure, Feature Flags and Decision Making
- Client-side vs server-side assignment and GrowthBook/LaunchDarkly/Statsig patterns
- Exposure logging schemas, automated reporting and the Ship / Iterate / Rollback matrix
Other Courses in This Category
-
Expressive naming, single-responsibility functions, spotting code smells, safe refactoring moves, clean error handling and maintainable unit tests (TDD) — for developers writing readable, easy-to-change production code.
- Live Course
- 5 modules
-
End-to-end software testing lifecycle (STLC), deriving test cases from requirements, exploratory testing, API testing with Postman, database verification with SQL and clear bug reporting — hands-on software testing training.
- Live Course
- 5 modules
-
Catching requirement ambiguities early, BDD/Gherkin and Example Mapping, black-box design techniques (EP, BVA, decision tables, state transition, pairwise), risk-based prioritisation and traceability — for test analysts and BAs.
- Live Course
- 5 modules
-
Aligned with the ISTQB Certified Tester Foundation Level (CTFL v4.0) syllabus: the 7 testing principles, testing in SDLC/Agile, static reviews, black/white-box design techniques, test management and practice exam questions.
- Live Course
- 5 modules
-
Selenium WebDriver 4 (W3C protocol), resilient CSS/XPath locators, Explicit Wait synchronisation, Page Object Model, parallel execution on Selenium Grid/Docker and Allure reporting — for QA teams building enterprise web test automation.
- Live Course
- 5 modules
-
In-browser Cypress architecture, auto-waiting, network spying and stubbing with cy.intercept, fast login with cy.session, React/Vue component testing and CI parallelisation — for modern frontend and QA teams.
- Live Course
- 5 modules
Get in touch
