Cognitive

Part 8 · Society and Open Problems

Fairness

AI learns from a world that isn't always fair, so it can repeat unfairness unless people measure it and fix it.

3 min readBuilds on Machine Learning Paradigms, Generalization and Overfitting

The same threshold can lead to different outcomes for different groups.

Key concepts

  • Bias can enter through data, labels, proxies, design choices, and deployment.
  • Removing a sensitive field does not remove bias, because other data can act as a proxy.
  • Overall accuracy can hide poor performance for smaller groups.
  • Disaggregated evaluation measures results separately across groups.
  • Model cards and datasheets document limits and intended use.

Bias can enter at each stage of a system's life cycle.

Figure 51.1. Bias can enter at each stage of a system's life cycle.
An overall average can hide poor performance for one group.

Formal view and evidence

  • Established. When base rates differ and prediction is imperfect, calibration within groups and equal false positive and false negative rates cannot generally all hold at once.
  • Debated. Which fairness criterion is appropriate. The answer depends on context, values, law, and the harm at stake.
  • Open problem. Measuring fairness in open-ended generative systems, where outputs and harms are hard to enumerate.

Common misconception

Common belief

"Computers can't be biased because they use math."

In practice

Models learn from human data and human choices.

Key takeaways

  • Bias can enter through data, labels, design, and deployment.
  • Evaluate performance separately across groups.
  • Fairness criteria can conflict, so choosing among them involves values.