Training, Testing, and Overfitting

The goal of machine learning is not to do well on data you already have — you know those answers already — but to do well on data you have never seen. A model that perfectly reproduces its training data has done nothing useful if it stumbles the moment a real customer, patient, or transaction arrives. This gap between memorising and genuinely learning is where most machine learning projects quietly fail, often while looking impressive on paper. Understanding train and test splits, overfitting, and the bias-variance trade-off is what lets you tell a model that has learned from one that has merely memorised, before it embarrasses you in production.

Generalization and the train/test split

Generalization is a model's ability to perform on new, unseen data. To measure it honestly, you hold out part of your data — a test set — and never let the model train on it. You train on the rest, then check performance on the held-out test set. Because the model has never seen those examples, its score there estimates how it will behave in the real world. Testing on training data measures memory, not learning, and flatters almost anything.

Overfitting and underfitting

Overfitting happens when a model learns the training data too well, including its noise and accidents. It scores brilliantly on training data and poorly on test data, because it memorised specifics that do not generalise. Underfitting is the opposite: the model is too simple to capture the real pattern, so it does poorly on both training and test data. You detect overfitting by the gap — a large difference between strong training performance and weak test performance is its signature.

The bias-variance tension

Every model balances two errors. A model with high bias is too rigid; it oversimplifies and underfits. A model with high variance is too flexible; it chases noise and overfits. Reducing one tends to raise the other, so the aim is a middle ground that captures the real signal without the noise. To tune this without touching the test set, practitioners hold out a further slice — a validation set — used to compare choices during development, keeping the test set pristine for one final, honest measurement.

A team builds a model to predict which loan applicants will repay. On their training data it reaches 99% accuracy — almost perfect. On the held-out test set it manages only 68%. That gap is the tell: the model memorised quirks of the specific applicants it trained on rather than the general traits of good borrowers. They simplify the model, forcing it to rely on broad patterns, and the numbers converge — 80% on training, 78% on test. Lower on paper, but trustworthy, because the two scores now agree.

Consider the reverse. A second team uses a model so simple it predicts that everyone repays. It scores 71% on training and 71% on test — the scores agree, so surely it generalises? No. The scores match because the model learned almost nothing; it underfits. Matching train and test scores is necessary but not sufficient. A good model needs the scores to agree and both to be high. Agreement alone can mean a model that has failed equally everywhere.

A well-documented pattern across machine learning is the model that scores highly in testing yet disappoints on real-world inputs. It recurs in image recognition, medical imaging, and predictive systems, and the usual cause is a mismatch between the data a model trained on and the data it later meets — a different hospital's scanners, unfamiliar lighting, changed user behaviour. The model generalised to its test set, but the test set did not represent the wider world. This is why a strong test score is a starting point, not a guarantee, and why careful teams monitor performance after deployment rather than declaring victory at launch.

• "High training accuracy means a good model." It may mean overfitting; the test score decides.
• "Overfitting means the model is bad." It means too complex for the data — often fixable, not worthless.
• "You can tune on the test set." Doing so leaks it into training; that is exactly what the validation set prevents.
• "A simpler model is always safer." Too simple underfits and misses the real pattern.

Given the training and test scores for several models, decide which one is overfitting, which is underfitting, and which generalises well — and say what the pattern of scores told you.

Think Like a Maester: Never judge a model by how well it fits the data it has already seen. The only score that matters is the one from data it has not.

1. What does generalization mean, and why is it the real goal?
2. Why must the test set be kept separate from training?
3. What pattern of scores signals overfitting?
4. How does underfitting differ from overfitting?
5. What is the validation set for, and why not just use the test set?

The purpose of a machine learning model is to generalise — to perform on data it has never seen — so its true grade comes from a held-out test set, never from the data it trained on. Overfitting, where a model memorises noise and excels only on training data, and underfitting, where it is too simple to learn the pattern, are the two failure modes; the bias-variance trade-off names the tension between them, and a validation set lets you navigate it without spoiling the final test. A high training score proves nothing on its own; a model earns trust only when it does well on data it was never shown.

• Types of Machine Learning
• Evaluating a Machine Learning Model