Cognitive

Part 8 · Society and Open Problems

Open Problems in AI

AI is still a young science, and its biggest questions are waiting for curious people to answer them.

3 min readBuilds on Generalization and Overfitting, Interpretability, AI Safety and Governance

Research spreads outward from what is known into what is not.

Key concepts

The major open questions:

  • Why do huge neural networks generalize so well?
  • Can AI learn from just a few examples?
  • Do these models truly understand?
  • How do we verify what a model will do?
  • What should people always decide for themselves?

Progress in the field moves between theory, experiments, and deployment.

Figure 53.1. Progress in the field moves between theory, experiments, and deployment.
Why large networks learn so well remains one of the field's central questions.

Research map

A research map, with one entry point per question:

QuestionStatusStart with
Theory of deep learning generalizationOpenChapter 9
Understanding and meaning in language modelsDebatedBender & Koller (2020)
World models and planningDebatedLeCun (2022)
Efficient skill acquisitionOpenChollet (2019)
Mechanistic explanation at frontier scaleOpenChapter 19
Scalable oversightOpenChapter 52
Robust instruction-data separationOpenChapter 30
Machine consciousness and moral statusDebatedButlin et al. (2023)
Measuring long-horizon autonomyOpenKwa et al. (2025)

Common misconception

Common belief

"The experts have AI all figured out."

In practice

Experts disagree about many of the field's most basic questions.

Key takeaways

  • Major questions about learning and understanding remain open.
  • Experts disagree about several fundamental issues.
  • Progress depends on theory, experiments, and real-world evidence.