Explaining and Predicting with Regression

An earlier module introduced correlation — whether two things move together. Regression goes further: it builds a model of the relationship, letting you both predict one variable from others and explain how they relate while accounting for several factors at once. It is one of the most used tools in all of data analysis, and one of the most misused.

From correlation to a model

Where correlation gives a single number for how two things move together, regression fits a line (or surface) that best describes how one variable, the outcome, depends on one or more others. That model can then estimate the outcome for new cases — a prediction — and summarise the relationship as coefficients (how much the outcome changes per unit of each input).

Multiple regression and 'controlling for'

Multiple regression includes several input variables together. This lets analysts estimate the relationship of one factor with the outcome while holding the others constant — often described as controlling for those factors. For example, estimating how house price relates to size while accounting for location and age. This is powerful but delicate: you can only control for what you measure and include.

The dangers

Regression tempts over-interpretation. Coefficients are not proof of causation. Leaving out an important variable can badly distort the others (omitted-variable bias). And a model tuned too closely to one dataset can overfit — fitting noise and predicting poorly on new data.

An analyst models exam scores using hours studied and hours slept. The regression estimates each factor's association with scores while accounting for the other. It might find that, holding sleep constant, each extra study hour is associated with a few more points. Useful — but it is still an association within this data and these variables, not a guarantee that forcing more study causes higher scores for everyone.

A classic failure is trusting a coefficient while ignoring what's missing. Suppose ice-cream sales 'predict' drownings in a naive regression. Add temperature — the lurking variable — and the ice-cream effect vanishes. The first model wasn't wrong arithmetically; it was wrong because it omitted the real driver. Regression is only as trustworthy as the variables and thinking behind it.

The method's very name comes from a nineteenth-century discovery by Francis Galton, who studied how the heights of children related to their parents. He noticed that exceptionally tall parents tended to have children who were tall but, on average, closer to the population mean than the parents were — and likewise for short parents. He called this pull toward the average "regression toward the mean," and the line he drew to describe the parent-child relationship became the ancestor of modern regression analysis. The episode carries two lessons: regression grew from a real attempt to describe how variables relate, and 'regression to the mean' — extreme cases tending to be followed by less extreme ones — is itself a pervasive statistical trap, often mistaken for a real effect when a treatment merely coincided with a natural return toward average.

• "Regression proves what causes what." It models association; causation needs more (design, theory).
• "Controlling for factors removes all confounding." Only for variables you measured and included.
• "A model that fits the data perfectly is best." Overfit models capture noise and predict poorly on new data.
• "Regression and correlation are the same." Regression builds a predictive/explanatory model, not just a co-movement number.

Find a claim like 'X predicts Y' from a study. Brainstorm one important variable that might have been left out. How could including it change the story?

Think Like a Maester: A regression is only as honest as the variables it includes — always ask what was left out.

1. How does regression go beyond simple correlation?
2. What does 'controlling for' a variable mean, and what is its limit?
3. What is omitted-variable bias?
4. What is overfitting?
5. Where does the term 'regression' come from, and what trap does 'regression to the mean' name?

Regression models how an outcome depends on one or more variables, enabling prediction and explanation, and multiple regression estimates each factor's relationship while controlling for others you include. It is powerful but easily misused: coefficients are not causation, omitted variables distort results, and overfit models predict poorly. The method — and the cautionary idea of 'regression to the mean' — traces to Galton's study of parent-child heights, a reminder to always ask what a model leaves out.

• Correlation and Regression
• Comparing Many Groups: ANOVA