Cognitive

Part 8 · Society and Open Problems

AI Safety and Governance

As AI grows more capable, we need ways to make sure it pursues what people actually intend, and rules for how it is used.

3 min readBuilds on Reinforcement Learning, Instruction Tuning and RLHF, Interpretability

Optimizing a measure harder does not always bring more of what people intended.

Key concepts

  • Alignment means a system's behavior matches human intentions.
  • Specification gaming satisfies the stated objective while missing the real goal.
  • Goal misgeneralization happens when a capable system pursues the wrong goal in new situations.
  • Oversight includes evaluations, monitoring, and the ability to intervene.
  • Governance includes laws, standards, audits, and incident reporting.

Specification gaming. A system can maximize its measure while missing the real objective.

Figure 52.1. Specification gaming. A system can maximize its measure while missing the real objective.
Safety work continues across the whole life cycle of a system.

Formal view and evidence

  • Established. Specification gaming is widely documented. Goal misgeneralization has been demonstrated in controlled environments. The EU AI Act, Regulation (EU) 2024/1689, establishes risk-based obligations.
  • Debated. The probability and timing of severe AI risks, and the balance between open release and restricted access.
  • Open problem. Scalable oversight of systems more capable than their evaluators, detecting deceptive behavior, and verifying alignment rather than merely observing good behavior.

Common misconception

Common belief

"AI safety is only about robots taking over."

In practice

It includes today's practical problems, such as reliability, misuse, oversight, and accountability, as well as long-term risks.

Key takeaways

  • Alignment concerns whether behavior matches human intent.
  • Good test behavior does not prove good behavior everywhere.
  • Technical safeguards and governance work together.

Explore next