Part 8 · Society and Open Problems
Fairness
AI learns from a world that isn't always fair, so it can repeat unfairness unless people measure it and fix it.
Key concepts
- Bias can enter through data, labels, proxies, design choices, and deployment.
- Removing a sensitive field does not remove bias, because other data can act as a proxy.
- Overall accuracy can hide poor performance for smaller groups.
- Disaggregated evaluation measures results separately across groups.
- Model cards and datasheets document limits and intended use.
Bias can enter at each stage of a system's life cycle.
Formal view and evidence
- Established. When base rates differ and prediction is imperfect, calibration within groups and equal false positive and false negative rates cannot generally all hold at once.
- Debated. Which fairness criterion is appropriate. The answer depends on context, values, law, and the harm at stake.
- Open problem. Measuring fairness in open-ended generative systems, where outputs and harms are hard to enumerate.
Common misconception
Common belief
"Computers can't be biased because they use math."
In practice
Models learn from human data and human choices.
Key takeaways
- Bias can enter through data, labels, design, and deployment.
- Evaluate performance separately across groups.
- Fairness criteria can conflict, so choosing among them involves values.