From Data to Predictions

Every recommendation you receive, credit decision you face, and delivery estimate you trust rests on a model built from past data. The promise is simple: find a pattern in what already happened, then use it to anticipate what has not. When this works, it saves money, time, and effort at a scale no human could match.

But the same machinery can fool its makers. A model can fit yesterday's data almost perfectly and still be useless tomorrow, because it learned the accidents of the sample rather than the signal beneath it. Knowing how to tell a genuine pattern from a memorised fluke is the difference between a forecast you can act on and a confident number that quietly misleads you.

Training on the past

Building a predictive model means feeding it examples where the answer is already known: houses with their sale prices, emails already labelled spam or not. You then let the model adjust until its guesses match those answers. This fitting step is called training, and the examples are the training data. The hope is that the relationship it discovers, say between size and price, will hold for cases it has not yet seen.

Overfitting: memorising the noise

Real data carries two things at once: signal, the true underlying pattern, and noise, the random wobble specific to this particular sample. A model that is too flexible can bend to fit every last wobble, scoring almost perfectly on the training data. This is overfitting. It has memorised the noise, mistaking accident for law, and its apparent accuracy is an illusion that evaporates the moment new data arrives.

Judging on data it has not seen

The cure is discipline about evaluation. Before training, you set aside part of the data as a held-out test set and never let the model learn from it. You train on the rest, then check performance on that untouched portion, which stands in for the future. A model that shines on training data but stumbles on the test set is overfit. Only performance on unseen data tells you whether it has learned anything real.

Suppose you want to predict exam scores from hours studied, using twenty past students. A simple straight line captures the obvious trend: more study, higher scores, with some scatter. Now imagine you instead fit a wildly wiggly curve that threads through all twenty points exactly, with zero error. On the training data it looks perfect. But hand it a twenty-first student and it lurches to an absurd prediction, because the wiggles were chasing noise, not signal. Split the data first, training on fifteen and testing on five, and the straight line wins on the five held-out students, exposing the wiggly curve for what it is.

More complexity is not always overfitting, and a simple model is not automatically honest. A model can also underfit: a straight line forced onto data that genuinely curves will miss the real pattern and predict poorly on both training and test data. The goal is not minimum complexity but the right complexity, judged the same way in every case, by performance on data the model was never shown. Held-out testing is what distinguishes the two failures.

In October 2006 Netflix launched a public competition offering one million dollars to any team that could improve its Cinematch recommendation system by ten percent, measured by how accurately it predicted user movie ratings. Competitors received a training set of roughly 100 million past ratings. Crucially, the final scoring used ratings the teams never saw: the public leaderboard was computed on one hidden slice of data, and the prize on another slice kept secret, specifically so that teams could not tune their way to victory by peeking. This design is train-and-test discipline written straight into the rules.

The prize was won in 2009 by a team called BellKor's Pragmatic Chaos, which blended many models together. Yet there is a well-documented twist: Netflix reported that it never put the full winning solution into production, because the elaborate ensemble was too complex and costly to run for the modest real-world gain it offered. The competition rewarded squeezing out accuracy on a fixed test set; the business needed something maintainable. Both lessons matter: test on unseen data, and remember that a leaderboard score is not the whole of usefulness.

• 'High accuracy on the data we have means a good model.' Only accuracy on unseen data counts; training scores can be memorised noise.
• 'A more complex model is always better.' Extra flexibility often fits noise, hurting predictions on new cases.
• 'Overfitting means the model has a bug.' It is a modelling failure, not a coding error; the model did exactly what it was told, too well.
• 'Once tested, a model stays accurate forever.' The world drifts, so models decay and need re-checking against fresh data.

You are shown two models fitted to the same past sales data, along with their scores. Decide which will predict next month better, using only how each performs on held-out data rather than how neatly it fits the past.

Think Like a Maester: A model earns trust only on data it has never met; anything less is memory dressed up as prediction.

1. What is the difference between training data and a held-out test set?
2. In plain terms, what does it mean for a model to overfit?
3. Why can a model score perfectly on training data yet fail on new cases?
4. How did the Netflix Prize use unseen data to score competitors fairly?
5. Why is the most complex model not automatically the best one?

A predictive model learns a pattern from past data and applies it to the future. The central danger is overfitting: a flexible model can memorise the random noise in its training sample, looking flawless on old data while failing on new. The remedy is to hold out data the model never sees and judge it there, because only unseen data reveals whether it learned real signal. The Netflix Prize built this discipline into its rules, and its aftermath adds a second lesson: a model must be both accurate on fresh data and usable in the world it was built for.

• Forecasting and Prediction
• Communicating with Data