Cognitive

Part 3 · Language Models

Large Language Models

A large language model learns to predict the next token so well that it can explain, translate, code, and converse.

3 min readBuilds on The Transformer

A language model repeatedly predicts a distribution over the next token.

Key concepts

  • Pre-training teaches next-token prediction across enormous amounts of text and code.
  • Knowledge is stored in the model's weights, not in a database of documents.
  • The model's knowledge stops at a training cutoff, unless tools provide fresher information.
  • More data, compute, and parameters have generally improved capability, with rising costs.
  • Open-weight models publish their weights. Closed models are accessed through a product or API.

Pre-training repeats one objective at enormous scale: predict the next token.

Figure 16.1. Pre-training repeats one objective at enormous scale: predict the next token.
Empirical scaling laws relate model loss to available compute.

Formal view and evidence

  • Established. Pre-training minimizes the average negative log-likelihood −1T∑tlog⁡pθ(xt∣x<t). Empirical scaling laws relate loss to parameters, data, and compute. Chinchilla found earlier large models were undertrained, and that parameters and tokens should scale roughly together for a fixed compute budget.
  • Debated. Whether apparently sudden "emergent" abilities are real or partly an artifact of metrics. Whether statistical language modeling can amount to understanding is also contested.
  • Open problem. How far scaling continues, and how to attribute behavior to specific training data.

Common misconception

Common belief

"The model looks up answers in a giant database."

In practice

It generates from learned weights, which is why it can sound fluent and still be wrong.

Key takeaways

  • Pre-training minimizes error when predicting the next token.
  • Knowledge is stored in weights rather than in a document database.
  • Capability has improved with scale, but scale also raises cost.

Explore next