Amelia D.
Sample Guardrails saved us from a bad launch
Our variant won on sign ups but refunds rose. Without the guardrail lesson we would have shipped it.
A/B testing for business: design, read and act on experiments
Plan experiments with clear hypotheses and sample sizes, read p values and confidence intervals correctly, and avoid the classic traps
Created by Meera Pillai
- Updated September 2026
- English
- Intermediate
What you will learn
- Write hypotheses that connect a change to a metric and a decision
- Choose primary and guardrail metrics for an experiment
- Calculate sample size and duration from baseline and effect size
- Interpret p values and confidence intervals correctly
- Detect sample ratio mismatch and other validity problems
- Avoid peeking, multiple comparison and novelty traps
- Decide when an experiment is the wrong tool
Course content
Thinking in experiments
Planning the test
- Article: Baseline rate and minimum detectable effect 9 min
- Article: Sample size and statistical power 11 min
- Article: Test duration and weekly cycles 8 min
- Article: The experiment plan document 8 min
Reading results correctly
- Article: p values in plain language 10 min
- Article: Confidence intervals 10 min
- Article: Statistical versus practical significance 8 min
- Article: Sample ratio mismatch and data checks 9 min
Traps that fool smart teams
- Article: The peeking problem 9 min
- Article: Multiple comparisons and segments 9 min
- Article: Novelty and primacy effects 7 min
- Article: Interference between groups 8 min
Beyond the simple A/B test
- Article: A/B/n and multivariate tests 8 min
- Article: Sequential testing 9 min
- Article: Bayesian approaches 9 min
- Article: When not to run an A/B test 8 min
Running an experimentation programme
- Article: Choosing a testing tool 8 min
- Article: Building a review culture 8 min
- Article: Documenting and sharing learnings 7 min
- Quiz: Final check 6 min
Requirements
- Comfort with percentages and basic averages
- Access to a spreadsheet tool
Description
A/B tests promise a simple answer: did the change work? In practice, many tests are stopped too early, measured with the wrong metric or read in ways the statistics do not support. The result is confident decisions based on noise. This course teaches you to run experiments that produce trustworthy answers.
What you will learn
We use a running example from an online subscription business testing changes to its pricing page and onboarding emails. You will plan, analyse and decide on several experiments with realistic data.
- Writing hypotheses linked to a real decision
- Choosing primary and guardrail metrics
- Calculating sample size and test duration before you start
- Reading p values and confidence intervals in plain language
- Checking for sample ratio mismatch and other data problems
- Avoiding peeking, multiple comparison and novelty traps
- When to use sequential, Bayesian or multivariate approaches
- Building an experimentation habit across a team
How much maths is involved
You need to be comfortable with percentages and basic averages. Every statistical idea is explained with intuition first and a formula only where it helps. Calculations are done with free online calculators and a provided spreadsheet, so you can focus on judgement rather than arithmetic.
Who it is for
Product managers, marketers, growth teams and analysts who run or interpret experiments. If you have ever been asked "is this result significant?" and felt unsure, this course is for you. The concepts apply in any testing tool, and we look briefly at common platforms such as Optimizely, VWO, Statsig, GrowthBook and PostHog.
Who this course is for
- Product managers and marketers who run experiments
- Analysts asked to interpret test results
- Founders and growth teams deciding on product changes
Instructor
Meera Pillai
Business analyst teaching metrics, dashboards and experiments for people who make decisions
- 4.4 instructor rating
- 23 reviews
- 9,950 students
- 3 courses
- Sample
I am a business analyst who has spent years building reporting for sales, marketing and operations teams. Most of that work is less about clever charts and more about agreeing on definitions, cleaning data and making a dashboard that people open every Monday.
I teach analytics for business people and early career analysts: how to choose metrics, build a KPI tree, design honest dashboards in tools like Looker Studio, and run A/B tests without fooling yourself with the statistics.
I avoid jargon where I can and explain the maths where it matters. You will work with realistic sample datasets and leave with reports and templates you can adapt to your own team.
Student reviews
Leonie F.
Sample Finally understand p values
I have been reporting p values for two years without really understanding them. The plain language lesson fixed that.
Rafael G.
Sample Good balance of maths and intuition
Enough statistics to be correct without drowning in formulas. The spreadsheet calculator helped.
Sean M.
Sample The peeking simulation was eye opening
Seeing an A/A test appear to win made our team stop checking results every morning.
Clara S.
Sample Changed how we report results
We now share confidence intervals instead of just winner or loser. Leadership finds it clearer.
Parisa N.
Sample Strong planning section
Sample size and the plan document are now required before any test in our team. Clear and practical.
Hiroshi K.
Sample Wanted more on Bayesian methods
The frequentist content is excellent. The Bayesian lesson felt short compared with the rest.
Babajide O.
Sample Useful for marketers
I run email and landing page tests. The when not to test lesson was refreshingly honest.
Keep exploring
All Analytics for Business courses
More in Analytics for Business
$69.99
Enroll now