AI Ethics & Bias

Why this belongs in an engineering track, not just an ethics elective#

A machine learning model doesn't invent its patterns from nothing — it learns them from data that reflects the world (and the world's history) exactly as it was recorded, biases included. When that data encodes historical discrimination, uneven representation, or a flawed measurement process, the model doesn't correct for it; it learns and often amplifies it, then applies it at scale and with an air of mathematical objectivity that can make the resulting bias harder to challenge than an obviously biased human decision would be. This is a core engineering concern, not an add-on: a biased model in production is a defect, the same way a security hole or a data-corrupting race condition is — one that fairness metrics and testing can, and should, catch before deployment, not after a headline.

Where bias actually enters a system#

• Historical bias — the data accurately reflects a world that was itself unfair. A model trained on decades of hiring decisions where a demographic group was systematically passed over will learn "don't hire this group" as a real statistical pattern in the data, even with no discriminatory intent anywhere in the model or its training process.
• Representation bias — some groups are simply underrepresented in the training data, so the model has too little signal to perform well for them, even absent any historical unfairness. A dataset of face photos skewed heavily toward lighter skin tones will train a model that's measurably less accurate on darker-skinned faces, purely from a data-collection gap.
• Measurement bias — the label or proxy used as "ground truth" doesn't actually measure what it claims to. A well-documented example: using "was re-arrested" as a proxy for "committed another crime" quietly bakes in whatever bias exists in policing and arrest patterns themselves, not just in underlying criminal behavior.
• Proxy variables — a model can reconstruct a legally or ethically protected attribute (race, gender, disability) indirectly through correlated features (zip code, name, school attended), even when that attribute is deliberately excluded from the training data outright.

Documented real-world failures#

These aren't hypothetical — they're some of the most-cited, best-documented cases in the field, and worth knowing by name:

• COMPAS recidivism scores. A 2016 ProPublica investigation into a widely used US criminal-risk-scoring tool found that Black defendants were substantially more likely to be incorrectly flagged as high risk of reoffending than white defendants, while white defendants were more likely to be incorrectly flagged as low risk — a racial disparity in the tool's error rates, even though race wasn't an input feature. The tool's vendor disputed the analysis using a different, also-valid fairness metric — and that disagreement is itself the textbook illustration of the fairness trade-offs discussed below.
• Amazon's scrapped recruiting tool. Reported by Reuters in 2018, Amazon built and then abandoned an internal AI resume-screening tool after discovering it had taught itself to penalize resumes containing the word "women's" (as in "women's chess club captain") and downgraded graduates of two all-women's colleges — because it had been trained on a decade of the company's past hiring decisions, a period during which the tech industry hired men at much higher rates, so the model learned that historical pattern as if it were a valid signal of a good hire.
• Gender Shades (Buolamwini & Gebru, 2018). An MIT/Stanford study of three commercial facial-analysis systems found error rates for classifying gender as low as roughly 1% for lighter-skinned men, but as high as roughly 35% for darker-skinned women — a gap traced largely back to training and benchmark datasets that skewed heavily toward lighter-skinned faces. The study directly prompted several vendors to publicly revise their training data and re-test.
• Google Photos' 2015 mislabeling incident. An image-recognition feature mislabeled photos of Black people with an offensive category, a failure widely attributed to inadequately diverse training and testing data for the relevant categories. The company's fix at the time, as reported by multiple outlets, was to simply disable the offending label entirely rather than fix the underlying classification — illustrating how hard some of these failures are to properly correct rather than paper over.

Fairness trade-offs: there is no single "unbiased"#

A genuinely uncomfortable mathematical result underlies a lot of this: several intuitively reasonable definitions of "fair" cannot all be satisfied simultaneously except in narrow special cases. Two of the most common:

• Demographic parity — the model should flag/select each group at the same rate, regardless of the group's true underlying rate of the outcome being predicted.
• Equalized odds — the model's error rates (false positive rate, false negative rate) should be equal across groups, regardless of each group's selection rate.

When the base rate of the outcome genuinely differs between groups in the data, satisfying demographic parity and equalized odds at the same time is, in general, mathematically impossible — this is exactly the disagreement at the center of the COMPAS controversy above: the tool's defenders pointed to one fairness metric it satisfied reasonably well, and its critics pointed to a different, also-legitimate one it failed. Building a "fair" system, in practice, means an explicit, documented choice about which notion of fairness matters most for a given application — not a checkbox that can be satisfied generically for every system at once.

Fairness definition	What it equalizes across groups	Can conflict with
Demographic parity	Selection/flag rate	Equalized odds, when base rates genuinely differ
Equalized odds	False positive and false negative rates	Demographic parity, when base rates genuinely differ
Predictive parity	Precision (accuracy of positive predictions)	Both of the above, in many realistic cases

Mitigation approaches#

• Audit training data for representation gaps before training, not after deployment — checking whether every relevant group is adequately represented in both the training and evaluation data.
• Measure performance per subgroup, not just in aggregate — an overall accuracy number can hide a large gap for a specific subgroup, exactly as it did in the Gender Shades study.
• Choose a fairness metric deliberately and document the choice — since multiple definitions can conflict, teams need to explicitly decide (and be able to explain) which one the system is optimized for, rather than assuming "fair" is a single unambiguous target.
• Use model cards and datasheets — short, standardized documents (a practice proposed in AI research around 2018–2019) that describe a model's or dataset's intended use, known limitations, and evaluated performance across subgroups, so downstream users don't have to rediscover these limitations the hard way.
• Keep a human in the loop for high-stakes decisions — hiring, lending, criminal justice, and medical triage are exactly the domains where an automated system's output should be a recommendation subject to human review, not a fully automated final decision.

Common mistakes#

• Assuming removing a protected attribute (race, gender) from the training data makes a model unbiased — proxy variables (zip code, name, school) routinely let a model reconstruct the excluded attribute anyway, indirectly.
• Treating "fair" as a single, objective, universally agreed-upon property a model either has or lacks — different, individually reasonable fairness definitions can mathematically conflict, so any real system embodies an explicit choice, not a neutral default.
• Auditing only overall accuracy and never breaking results down by subgroup — this is exactly how the failures above went undetected until independent researchers or journalists checked subgroup performance specifically.