Comparing Groups and A/B Testing

Almost every decision worth making is a comparison. Does the new layout beat the old one? Does the medication help more than a sugar pill? Does the training raise output? The trap is that groups differ for countless reasons at once, so when you see a gap between them, you rarely know which reason caused it.

The fix is the controlled comparison: change one thing, hold everything else as equal as you can, and watch what happens. An A/B test is simply this idea run on real users. Learning to build one, and to read one honestly, protects you from the most common mistake in data, mistaking a coincidence, or a hidden difference between the groups, for a genuine effect.

Hold everything else equal

The reason a fair comparison is hard is confounding: some other factor differs between your groups alongside the one you care about. If you show the new website only to weekend visitors and the old one to weekday visitors, any difference could be the day, not the design. A good comparison isolates a single change so the effect you measure belongs to that change.

Random assignment does the isolating

You cannot list, let alone match, every way people differ. Random assignment solves this by shuffling people into group A or group B by chance. On average, the two groups end up similar in every respect, known and unknown, except the one thing you deliberately vary. That is the quiet genius of the A/B test and the randomized experiment behind it.

Sample size and the two significances

Small samples are noisy; a lead of a few clicks out of fifty means little. Larger samples make real differences visible and shrink the role of luck. But once samples are huge, tests can flag differences that are real yet trivially small. Statistical significance asks whether a difference is probably not chance. Practical significance asks whether it is big enough to bother acting on. You need both.

A newsletter team tests two subject lines. They randomly split their list: half receive line A, half receive line B, sent at the same time to the same kind of readers. Line A gets a 22 percent open rate, line B gets 20 percent, across fifty thousand recipients. Because the split was random and the sample is large, the two-point gap is unlikely to be noise, and two points of opens on a big list is worth keeping. They adopt line A. The design, not the guess, decided it.

A manager notices the team that used a new tool outperformed the team that did not, and credits the tool. But the teams were not randomly assigned: the new tool went to the most experienced staff, who chose to adopt it. Experience, not the tool, plausibly explains the gap. Without random assignment, the comparison cannot separate the two, and the confident conclusion is unearned.

Controlled online experiments are now standard practice at large technology firms, which run many such tests continuously to compare versions with live users. A widely reported early example is Google's testing of link colors around 2009: rather than let a designer pick, the company reportedly tried roughly forty-one gradations of blue on real traffic to see which drew more clicks. The winning shade was chosen by measured behavior, not opinion.

The logic is far older than the web. In the 1920s and 1930s the statistician Ronald Fisher, working on agricultural trials at the Rothamsted Experimental Station in England, formalized randomized controlled experiments in his 1935 book "The Design of Experiments." Randomly assigning treatments to plots let him credit differences in yield to the treatment rather than to the soil. An A/B test applies that same century-old logic to buttons and headlines.

• "A visible difference between groups proves the change caused it." Not without controlling for confounders.
• "You do not really need random assignment." Randomization is what balances the unknown differences.
• "Statistically significant means important." With big samples, tiny, useless differences can be significant.
• "A bigger sample can never mislead." It can flag real but trivial effects you should not act on.

Picture any two groups being compared, say, users of an app feature versus non-users. List three ways those groups might already differ before the feature is even considered. Each one is a reason random assignment exists.

Think Like a Maester: A difference between groups means nothing until you know the groups were alike to begin with.

1. What is a confounding variable, and why does it threaten a comparison?
2. How does random assignment make two groups comparable?
3. Why can a small sample make an A/B test unreliable?
4. What is the difference between statistical and practical significance?
5. Who formalized the randomized controlled experiment, and in which 1935 book?

To tell a real difference from noise, compare groups that differ in only one thing. Random assignment balances the unknowns so the effect belongs to your change, not to a hidden confounder. Watch sample size, and always ask both questions: is the difference real, and is it big enough to matter? From Fisher's field trials to modern A/B tests, that is the whole craft.

• Statistical Significance
• From Data to Predictions