Comparing Many Groups: ANOVA

Often you have not two groups but several — three fertilisers, four web designs, five schools — and you want to know whether they differ. You might think: just run a t-test on every pair. That seemingly reasonable idea creates a hidden trap, and avoiding it is why ANOVA (analysis of variance) exists.

The multiple-comparisons problem

Each statistical test carries a small chance of a false positive — flagging a difference that is really just chance. Run one test at the usual 5% threshold and that risk is modest. But run many tests, and the chances that at least one comes up falsely significant balloon. Comparing five groups pairwise means ten tests; the probability of a spurious 'finding' among them is uncomfortably high. This is the multiple-comparisons problem.

What ANOVA asks

ANOVA sidesteps this by asking a single overall question first: is there any real difference among all the group averages, or could the whole spread be chance? It does this cleverly, by comparing the variation between the groups with the variation within them. If between-group variation is large relative to within-group noise, that signals real differences somewhere.

After ANOVA

If ANOVA finds that some groups differ, follow-up (post-hoc) tests, adjusted to control the inflated error, identify which pairs differ. ANOVA is the gatekeeper; the post-hoc tests do the pinpointing.

A gardener tests four fertilisers on plant growth. Rather than six separate t-tests (inviting false positives), they run one ANOVA: is the variation between the four fertiliser averages bigger than the random variation among plants given the same fertiliser? If yes, something differs, and controlled follow-up tests reveal which fertilisers stand apart. One overall question, then targeted comparisons.

ANOVA is not a cure-all. It tells you that some difference exists, not which or how large — you still need post-hoc analysis and effect sizes. And, like the t-test, it assumes reasonable conditions; badly skewed data or wildly unequal spreads can mislead it. Running ANOVA does not license ignoring the data's shape.

Much of ANOVA and modern experimental design comes from Ronald Fisher, working at the Rothamsted agricultural research station in England in the 1920s and 1930s, where he needed to compare many crop treatments across variable fields. Fisher developed analysis of variance to separate real treatment effects from natural variation. He is also famous for a thought experiment (and reportedly a real test) known as "the lady tasting tea": a colleague claimed she could tell whether milk or tea was poured into the cup first. Fisher used it to work out how to design an experiment — how many cups, in what random order — so that her success could be judged against what chance alone would produce. The episode, described in his 1935 book "The Design of Experiments," captures the spirit of the whole toolkit: careful design plus the right test lets us tell a real effect from a lucky guess.

• "Just run a t-test on every pair." Many tests inflate the chance of false positives.
• "ANOVA tells you which groups differ." It flags that some difference exists; post-hoc tests find which.
• "A significant ANOVA means big or important differences." It shows some difference; size needs effect measures.
• "ANOVA needs no assumptions." It assumes reasonable conditions like the t-test.

Imagine comparing six groups pairwise. How many separate t-tests is that? (Hint: pairs of six.) Reflect on why that many tests inflates false positives — and why one ANOVA helps.

Think Like a Maester: When comparing many groups, ask the overall question first — running test after test is how chance sneaks in a false discovery.

1. What is the multiple-comparisons problem?
2. What single question does ANOVA ask?
3. How does ANOVA compare between-group and within-group variation?
4. What are post-hoc tests for?
5. Who developed ANOVA, and what does 'the lady tasting tea' illustrate?

Comparing many groups by running a t-test on every pair inflates the chance of false positives — the multiple-comparisons problem. ANOVA avoids this by first asking one overall question: is the variation between group averages larger than the random variation within groups? If so, controlled post-hoc tests identify which groups differ. Developed by Fisher at Rothamsted and illustrated by 'the lady tasting tea,' it embodies the toolkit's spirit: good design plus the right test distinguishes real effects from chance.

• Exploratory Data Analysis
• Beyond p-values: Effect Sizes and Intervals