Why test instead of guess

Most product and marketing changes are made on opinion: a manager likes the new headline, a designer is confident the new layout is clearer, a competitor did something similar. Opinions are often wrong, and even experienced teams find that a large share of their ideas do not improve the metrics they were meant to improve. An A/B test replaces opinion with evidence.

What an A/B test is

In an A/B test, you randomly split people into two groups. Group A sees the current version, the control. Group B sees the changed version, the variant. Because assignment is random, the groups are alike on average in every respect except the change. If the variant group behaves differently, beyond what chance would explain, the change is the most likely cause.

This is what makes experiments powerful. They establish cause and effect, which ordinary analytics cannot. A dashboard might show that customers who use a feature retain better, but those customers may simply be more engaged to begin with. An experiment removes that ambiguity.

Why intuition is not enough

• Effects are usually small. A good change might lift conversion from 4.0 to 4.3 percent. You cannot see that by eye in daily numbers, which vary more than that naturally.
• Many things change at once. Seasonality, marketing campaigns and news all move metrics. Before and after comparisons mix your change with everything else.
• Surprising results are common. Changes that everyone expects to help sometimes hurt, and small details sometimes matter more than big redesigns.

What tests cannot do

Experiments are not always possible or sensible:

• You need enough traffic. With a few hundred visitors a month, most tests will never reach a reliable answer.
• Some changes cannot be randomised fairly, such as a new price for existing customers that they would compare with each other.
• Long term effects like brand perception or retention over a year are hard to capture in a test of a few weeks.

We cover alternatives for these situations later in the course.

Our running example

We will work with a company that sells a subscription meal planning app. Its pricing page converts about 4 percent of visitors to trial sign ups. The team has ideas: a new headline, an annual plan discount shown more prominently and a shorter sign up form. Throughout the course we will plan, run and interpret experiments on these ideas.

Write down one change your team made recently based on opinion. Could it have been tested? What would you have measured?