Part 8 · Society and Open Problems
AI Safety and Governance
As AI grows more capable, we need ways to make sure it pursues what people actually intend, and rules for how it is used.
Key concepts
- Alignment means a system's behavior matches human intentions.
- Specification gaming satisfies the stated objective while missing the real goal.
- Goal misgeneralization happens when a capable system pursues the wrong goal in new situations.
- Oversight includes evaluations, monitoring, and the ability to intervene.
- Governance includes laws, standards, audits, and incident reporting.
Specification gaming. A system can maximize its measure while missing the real objective.
Formal view and evidence
- Established. Specification gaming is widely documented. Goal misgeneralization has been demonstrated in controlled environments. The EU AI Act, Regulation (EU) 2024/1689, establishes risk-based obligations.
- Debated. The probability and timing of severe AI risks, and the balance between open release and restricted access.
- Open problem. Scalable oversight of systems more capable than their evaluators, detecting deceptive behavior, and verifying alignment rather than merely observing good behavior.
Common misconception
Common belief
"AI safety is only about robots taking over."
In practice
It includes today's practical problems, such as reliability, misuse, oversight, and accountability, as well as long-term risks.
Key takeaways
- Alignment concerns whether behavior matches human intent.
- Good test behavior does not prove good behavior everywhere.
- Technical safeguards and governance work together.