Bias, Data, and the Limits of Machine Learning

Machine learning does not learn what is true or fair; it learns whatever patterns sit in the data it was given, measured against whatever objective it was told to optimize. If that data records past human decisions, past inequalities, or a narrow slice of the world, the model absorbs those features and reproduces them at scale. This is the plain meaning of "garbage in, garbage out." The stakes are highest where these systems are now deployed — hiring, lending, policing, healthcare — because a biased model does not look biased. It looks precise, fast, and neutral. Knowing the limits of what machine learning can know is the difference between using it as a tool and trusting it as an oracle.

Garbage in, garbage out

A model is a compressed summary of its training data. Skews in that data — who is over-represented, whose outcomes were recorded, what a label actually measured — become skews in the model. The objective matters just as much: optimizing for "past hires" bakes in the assumption that yesterday's pattern is tomorrow's goal.

Correlation, not understanding

Machine learning finds statistical associations, not causes or meaning. It may notice that a word, a postcode, or a name correlates with an outcome and use it, with no notion of whether the link is fair, causal, or a historical accident. The model knows what tends to go with what, never why.

Distribution shift

Models assume the future resembles the training data. When the world changes — a new market, a pandemic, a different population — the learned patterns can quietly stop holding. The model still outputs confident numbers; they are just increasingly wrong. This silent failure is called distribution shift.

A bank trains a loan-default model on ten years of its own lending records. Because it historically lent less to certain neighborhoods, those areas have few examples of successful repayment. The model learns to associate the neighborhood with risk — not because residents default more, but because the data barely contains their successes. Deployed, it denies loans there, generates no new counter-evidence, and so confirms its own bias. The objective was met perfectly; the outcome was still unjust.

Not every gap is a bias to be fixed by more data. A weather model trained on a stable climate is not "biased" against future heatwaves — it is being asked about a world that no longer matches its training. Here the failure is distribution shift, not prejudice. Before blaming the data's fairness, ask whether the world itself has moved beyond what the model ever saw.

Around 2018, Reuters reported that Amazon had abandoned an experimental recruiting tool. Trained on roughly ten years of resumes submitted to the company — mostly from men — the system learned to penalize CVs containing the word "women's" and downgraded graduates of two all-women's colleges. Amazon says the tool was never used to evaluate real candidates and was scrapped. Separately, ProPublica's 2016 investigation of COMPAS, a recidivism-risk tool used in some US courts, found that Black defendants were more often incorrectly flagged as high-risk, while white defendants were more often incorrectly flagged as low-risk. Northpointe, its maker, disputed the analysis, and researchers noted the tool satisfied different, mathematically competing definitions of fairness. Both cases share one root: models trained on human decisions inherit human patterns.

• "The algorithm is objective." It reflects its data and objective; neither is neutral.
• "Remove the sensitive field and bias disappears." Proxies like postcode or vocabulary carry the same signal.
• "More data always fixes it." More of the same skewed data entrenches the skew.
• "High accuracy means it is fair." A model can be accurate overall and unequal across groups.

You are handed a hiring model's inputs, objective, and error rates by group. Decide whether to deploy it, patch it, or reject it — and justify which limit of machine learning drove your call.

Think Like a Maester: A model cannot know more than its data, or want anything other than its objective. Before trusting a prediction, ask what it was trained on and what it was told to optimize.

1. What does "garbage in, garbage out" mean for machine learning?
2. Why can a model be biased even when it never sees a person's race or gender?
3. What is distribution shift, and why is it a silent failure?
4. Why does high overall accuracy not guarantee fairness across groups?
5. What did the Amazon and COMPAS cases have in common at their root?

Machine learning learns patterns, not truth or fairness, and it inherits every skew and gap in its training data and objective. It finds correlations without understanding, and it assumes tomorrow will resemble the data it was trained on — an assumption that distribution shift can quietly break. Real systems, from Amazon's scrapped recruiter to the disputed COMPAS tool, show that inheriting human decisions means inheriting human bias. The closing skill of this module is knowing what machine learning can decide well, and refusing to let it decide what it cannot.

• AI Myths vs. Facts
• Critical Thinking