Bias and Fairness in AI

AI systems increasingly help decide who gets a loan, a job interview, or a medical follow-up. When these systems are unfair, the harm is not abstract: it falls on real people, often those already disadvantaged, and it can do so quietly and at scale.

Earlier you met the idea that data can carry bias. Here we go further. Bias is not a single flaw to be patched, and 'fairness' is not one thing. Reasonable, well-intentioned people disagree about what a fair system even means, and mathematics shows that some of these goals are genuinely incompatible.

Where bias comes from

Bias can enter at every stage. Data may reflect an unequal world: if past hiring favored one group, a model trained on it learns that pattern. Labels encode human judgments that may themselves be skewed, or measure a proxy (like 'arrested') instead of the thing we care about (like 'committed a crime'). Design choices, such as which features to include or how to weight errors, embed values. Finally, deployment matters: a tool accurate in one setting can mislead when used on a different population or for a purpose it was never built for.

Competing definitions of fairness

Suppose a model scores loan applicants. One notion of fairness asks for calibration: among everyone assigned a 70% repayment score, about 70% should repay, regardless of group. Another asks for equal error rates: the system should wrongly deny creditworthy applicants at the same rate across groups. Both sound fair. Both are defensible. But when base rates differ between groups, researchers have proven you generally cannot satisfy both at once.

Detecting and reducing bias

Practitioners audit systems by measuring outcomes across groups, testing on diverse data, and inspecting errors. Mitigation can happen before training (rebalancing data), during training (adding fairness constraints), or after (adjusting thresholds). Each choice trades one goal against another, so the honest question is not 'is it fair?' but 'fair in which sense, and at what cost?'

Imagine two groups. In Group A, 50% of applicants actually repay; in Group B, 30% do. A single, well-calibrated scoring rule is applied to both. Because the underlying repayment rates differ, the calibrated rule will produce different false-positive and false-negative rates across the groups. To equalize those error rates, you would have to abandon calibration or use different thresholds per group. The trade-off is not a bug to be fixed; it is a mathematical consequence of differing base rates.

It is tempting to think that removing the sensitive attribute, say race or gender, makes a model fair. It usually does not. Other features act as proxies: a postal code can stand in for race, a first name for gender. A model blind to the label can still reproduce the disparity through correlated variables, and hiding the attribute also makes bias harder to measure. Fairness by deletion is often fairness by concealment.

In 2016, the newsroom ProPublica (Julia Angwin and colleagues) analyzed COMPAS, a risk-assessment tool by Northpointe used in some US courts. They reported that Black defendants who did not reoffend were more often labeled high-risk, while white defendants who did reoffend were more often labeled low-risk. Northpointe responded that the tool was equally calibrated across races. Researchers including Alexandra Chouldechova and Jon Kleinberg then showed both sides could be right: calibration and equal error rates cannot generally coexist when base rates differ. The dispute became a landmark illustration of incompatible fairness definitions.

Separately, in 2018 Joy Buolamwini and Timnit Gebru published 'Gender Shades', auditing three commercial gender-classification systems. Accuracy was near-perfect for lighter-skinned men but far worse for darker-skinned women, with error rates above 30% for one system. The uneven training data and testing had left whole groups poorly served.

• 'The algorithm is neutral because it is just math.' Choices about data, labels, and objectives carry human values.
• 'Removing race or gender makes a model fair.' Correlated proxies can preserve the bias while hiding it.
• 'There is one correct fairness metric.' Several exist, and some are provably incompatible.
• 'More data always fixes bias.' More of the same skewed data can entrench it.

Using the loan scenario, choose whether to enforce calibration or equal error rates, then read off which group bears more false denials under each. Notice that improving one metric worsens another, and write down which trade-off you would defend and to whom.

Think Like a Maester: Before asking whether a system is fair, ask 'fair in which sense, measured how, and at whose expense?'

1. Name three distinct stages where bias can enter an AI system.
2. Why can calibration and equal error rates conflict?
3. Why does deleting a sensitive attribute often fail to remove bias?
4. What did the COMPAS debate reveal about fairness definitions?
5. What did Gender Shades find about commercial facial-analysis accuracy?

AI bias arises across data, labels, design, and deployment, not from one isolated defect. Fairness is plural: calibration and equalized error rates are each reasonable yet can be mathematically incompatible when groups differ in base rates. The COMPAS dispute and the Gender Shades audit show these tensions in the real world. Detecting and reducing bias is possible and important, but it means choosing trade-offs deliberately rather than pretending a single, cost-free fix exists.

• Bias, Data, and the Limits of Machine Learning
• AI, Jobs, and the Economy